Adversarial attacks on convolutional neural networks (CNN) have gained significant attention and there have been active research efforts on defense mechanisms. Stochastic input transformation methods have been proposed, where the idea is to recover the image from adversarial attack by random transformation, and to take the majority vote as consensus among the random samples. However, the transformation improves the accuracy on adversarial images at the expense of the accuracy on clean images. While it is intuitive that the accuracy on clean images would deteriorate, the exact mechanism in which how this occurs is unclear. In this paper, we study the distribution of softmax induced by stochastic transformations. We observe that with random transformations on the clean images, although the mass of the softmax distribution could shift to the wrong class, the resulting distribution of softmax could be used to correct the prediction. Furthermore, on the adversarial counterparts, with the image transformation, the resulting shapes of the distribution of softmax are similar to the distributions from the clean images. With these observations, we propose a method to improve existing transformation-based defenses. We train a separate lightweight distribution classifier to recognize distinct features in the distributions of softmax outputs of transformed images. Our empirical studies show that our distribution classifier, by training on distributions obtained from clean images only, outperforms majority voting for both clean and adversarial images. Our method is generic and can be integrated with existing transformation-based defenses.

===== Post Rebuttal =====
The authors addressed almost all of my comments. I think the revised paper is of higher quality and clarity as a result. Figure 3 is the main remaining concern. However,  given the listed strengths of the paper, I am happy to change my rating to weak accept .
===== Summary =====
One strategy for defense against adversarial attacks is to perturb the input sample multiple times and then aggregate their corresponding outputs into a single output. One recent variant of this method (Prakash et al. 2018) shows using this type of defense mechanism degrades the accuracy on clean (unperturbed) inputs. This paper tries to mitigate the accuracy drop on clean inputs. It does so by learning an additional distribution classifier which takes as input the distribution of perturbed samples’ outputs. The method is trained on MNIST, CIFAR10, CIFAR100 and using 4 different adversarial attacks. It shows significant improvement in several cases over standard input-perturbation methods.
===== Strengths and Weaknesses =====
+ The paper picks an important problem, proposes a simple technique and achieves significant improvements in certain cases.
+ The experiments are done on 3 datasets and 4 different adversarial attack techniques.
+ The final experiment on end-to-end attack is interesting.
Concerns regarding the main paper’s description
- From Figure 1, it seems that the distribution classifier does a binary classification for the distribution of each class separately. If this is the case, how is the final classification done (using the binomial distribution per class)? The figure suggests that the maximum probability of the binary classifications (bus for yellow class with 0.8 score) is taken. This would be strange since they are not optimized to compete with each other (e.g. as in a more proper cross entropy). In any case, the details of the distribution classifier is missing from the paper, and section 4 refers to the figure. The details should come in the main text using precise language and math formulation.
- page 4 is very hard to follow since it tries to give a textual description to mathematical concepts. I think it is important to provide the mathematical definition such that 1) it is easier to follow 2) the concepts are unambiguously defined. This includes a definition of softmax distribution, joint distribution of output and transformation, marginalized distributions, clean intra-class distance, adversarial intra-class distance, and clean-adversarial inter-class distance.
- page 4: it seems it is implied in the definition of “clean” images that these images are classified “correctly” by the network. Otherwise, in Figure 2.a, the clean row is uninterpretable not knowing whether the clean image was classified correctly or not.
- page 4,5: what is a “transformation magnitude”? It could refer to both the number of transformations and also the number of perturbed pixels. The behavior in Figure 3 suggests that the latter should be the case, but it’s important to disambiguate.
- figure 3 is counterintuitive. The support (possible distributions) should normally live on a simplex (due the sum-to-one probability constraint). In figure 3, it seems there is support in the full unit hypercube. In particular, we can see density along the horizontal/vertical line of 1-density for one class (i.e. non-zero density for others in this case)
- page 6: “[...] to train a distribution classifier on the distributions obtained from clean images only [...]“. I cannot follow the argument at the end of the first paragraph that led to the proposed *training on clean images only*. I understand that from the qualitative demonstration of 8 different examples in Figure 4 the distributions of perturbed adversarial examples *might be* redundant to the perturbed wrongly-classified clean examples. However, even if it is redundant, why should it be harmful to train the distribution classifier with adversarial examples as well as clean examples?
Regarding Experiments:
- why is the evaluation done only based on accuracy of *correctly-classified* clean images and the recovery of *wrongly-classified* adversarial images? Couldn’t it be that the baseline (majority voting) does a better job than the distribution classifier on the “wrongly-classified” clean images and the “correctly-classified” adversarial images?
- I think it's still interesting and informative to train the distribution classifier using both clean and perturbed inputs to compare the performance.
- page 6: why is there a need for applying a kernel density estimator followed by a discretization (e.g. for instance instead of a simple histogram)?
===== Final Decision =====
My current “weak reject” rating is primarily based on the unclarity of the main paper and secondary regarding the concerns for experiments.
===== Points of Improvement =====
I believe the paper will become stronger if the proposal is described and motivated more formally.


How to generate descriptions from structured data organized in tables? Existing approaches using neural encoder-decoder models often suffer from lacking diversity. We claim that an open set of templates is crucial for enriching the phrase constructions and realizing varied generations. Learning such templates is prohibitive since it often requires a large paired <table,description> corpus, which is seldom available. This paper explores the problem of automatically learning reusable "templates" from paired and non-paired data. We propose the variational template machine (VTM), a novel method to generate text descriptions from data tables. Our contributions include: a) we carefully devise a specific model architecture and losses to explicitly disentangle text template and semantic content information, in the latent spaces, and b) we utilize both small parallel data and large raw text without aligned tables to enrich the template learning. Experiments on datasets from a variety of different domains show that VTM is able generate more diversely while keeping a good fluency and quality.

This paper proposes Variational Template Machine (VTM), a generative model to generate textual descriptions from structured data (i.e., tables). VTM is derived from the variational autoencoder, where the input is a row entry from a table and the output is the text associated with this entry. The authors introduce two latent variables to model contents and templates. The content variable is conditioned on the table entry, and generates the textual output together with the template variable. The model is trained on both paired table-to-text examples as well as unpaired (text only) examples. Experiments on the Wiki and SpNLG datasets show that models generate diverse sentences, and the overall performance in terms of BLEU is only slightly below the best baseline Table2Seq model that does not generate diverse sentences. The results also show that additional losses for preserving contents and templates introduced by the authors play an important role in the overall model performance.
I have several questions regarding the experiments:
- For the Table2Seq baseline, how was the beam size chosen? Did it have any effect on the performance of the baseline model?
- Did the authors try other sampling methods for Table2Seq? (e.g., top-K or nucleus sampling)
- VTM is only able to achieve comparable performance to Table2Seq in terms of BLEU after including the unlabeled corpus, especially on the Wiki dataset. A way to incorporate this unlabeled data to Table2Seq is by first pretraining the LSTM generator on it before training it on pairwise data (or in parallel). How would this baseline model perform in comparison to VTM?
- In the conclusion section, the authors mentioned that VTM outperforms VAE both in terms of diversity and generation quality. What does this VAE model refer to? The experiments show that VTM is comparable to Table2Seq in terms of quality and is better in terms of diversity.
Generating text from structured data is an interesting research area. However, I am not convinced that the proposed method is a significant development based on the results presented in the paper. There are also many grammatical errors in the paper (e.g., ... only enable to sample in the latent space ..., and many others), so I think the writing of the paper can be improved.


We present a simple and effective algorithm designed to address the covariate shift problem in imitation learning. It operates by training an ensemble of policies on the expert demonstration data, and using the variance of their predictions as a cost which is minimized with RL together with a supervised behavioral cloning cost. Unlike adversarial imitation methods, it uses a fixed reward function which is easy to optimize. We prove a regret bound for the algorithm in the tabular setting which is linear in the time horizon multiplied by a coefficient which we show to be low for certain problems in which behavioral cloning fails. We evaluate our algorithm empirically across multiple pixel-based Atari environments and continuous control tasks, and show that it matches or significantly outperforms behavioral cloning and generative adversarial imitation learning.

Summary of what the paper claims and contributes
---
This paper proposes a new interactive imitation learning algorithm to address the covariate shift problem in imitation learning. It explicitly seeks to avoid settings interactive expert feedback (e.g. DAgger). The method is straightforward: 1. First, learn an ensemble of policies via KL-based Behavior Cloning 2. Then, learn a new policy via a new objective that combines the original Behavior Cloning objective with a "disagreement" loss, formed by computing the expected variance of the ensemble evaluated on state-action trajectories under the new policy. The intuition for the method is that by learning an ensemble, it will have low variance on in-distribution demonstration data, and high variance on out-of-distribution other data; by encouraging the policy to seek regions of low variance, it should result in a policy that more closely matches the demonstrator's state visitation distribution than Behavior-Cloning alone. Analysis in the discrete finite case shows that the algorithm achieves regret linear in \kappa*T, where \kappa is an environment- and expert-dependent constant. The analysis is instantiated for a simple MDP, and experiments comparing their algorithm on this restricted environment provide some evidence that the bound is achievable in practice.
Further experiments on a variety of Atari environments and continuous-control tasks from OpenAI Gym also 1) demonstrates that their algorithm outperforms Behavior Cloning in these settings 2) usually approaches expert performance with a small number of demonstrations, and 3) also shows that the uncertainty cost improves over time, indicating the final policy learns to visit states where the ensemble agrees, and that while doing so, improves performance on the underlying task.
Evaluation
---
>Originality:
Are the tasks or methods new?
The method is new.
Is the work a novel combination of well-known techniques?
Yes.
Is it clear how this work differs from previous contributions?
Yes.
Is related work adequately cited?
There is some missing discussion of related works:
1. EnsembleDAgger (Menda 2018) also uses the variance of ensembles in Imitation Learning, but instead of using it to regularize on-policy learning, it uses it as an improved decision criterion by which to query an expert demonstrator.
2. Data as Demonstrator (Venkatraman 2015) uses on-policy learning to create "corrections" of time-series models (See their Fig 1), which is similar to this paper's intuition of seeking to push the learner back to places that are in-distribution of the expert demonstrations. That paper also achieves a linear regret bound under some assumptions.
>Quality:
Is the submission technically sound?
Mostly, although there are some issues:
1. Step 9 of the algorithm is ambiguous. What is the distribution of on-policy data that is fed into the cost? E.g. how many rollouts from the policy are collected?
2. Why is the clipped cost negative, as opposed to 0?
3. Why was a clipped cost used at all? This cost is different from that used in the theoretical analysis. Some justification and discussion is needed for why the new cost was used, and whether the analysis still applies when it's used.
4. Throughout most of the paper, p(\pi | \mathcal D) represents the model ensemble. However, no discussion was dedicated to what we should expect this distribution to look like in theory and in practice. It depends on how the ensemble is constructed / learned. A degenerate case would be if all models in the ensemble converged to the same local optima, in which case they would agree everywhere, nullifying the cost penalty. Discussion of what properties this distribution must satisfy is missing. It probably needs full support over the space of policies such that the optimal policy is nearly realizable (within \epsilon)?
5. \kappa is overloaded: A. it's used as a function B. it's used as the optimal value of that same function. Consider using different notation for one of the, e.g. \kappa^* for the optimum, or \gamma for the function. Furthermore, it might help to make \kappa's dependencies clearer, which would help illustrate its independence of T.
6. Example 1: the fact that the policy always starts at s_1 is missing from the description (at least, an equivalent assumption is made in Ross 2010)
7. Example 1: it's not clear that setting \mathcal U = \{s_1, s_2\} achieves the optimum of \kappa(\mathcal U). Discussion of this aspect is needed.
8. Example 1: The statement that the variance is equivalent to the variance of the uniform distribution seems to be a strong assumption about p(\pi | \mathcal D). This missing assumption is related to point 4. I mentioned above^
9. The paper is missing discussion for why the analysis would not immediately extend to continuous state and action spaces.
Are claims well supported by theoretical analysis or experimental results?
Yes, although the experimental results would be made stronger if related approaches were considered, e.g. Reddy 2019. Right now, there's just a single method of comparison -- BC.
Is this a complete piece of work or work in progress?
Seems complete.
Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?
I believe so -- noting that BC ended up performing similar in environments where there is less drift was a good addition.
>Clarity:
Is the submission clearly written?
Yes.
Is it well organized?
Yes.
Does it adequately inform the reader?
Yes.
>Significance:
Are the results important?
Yes.
Are others (researchers or practitioners) likely to use the ideas or build on them?
Yes.
Does the submission address a difficult task in a better way than previous work?
Yes.
Does it advance the state of the art in a demonstrable way?
Yes.
Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?
Unique theoretical approach.
Additional feedback
---
Sec 3: "The threshold q defines a normal range of uncertainty based on the demonstration data, and values outside of this range incur a negative cost". The logic of this statement is confusing. 1. It's not clear what "outside" means from the sentence alone (i.e. it should be "above"). 2. A single value doesn't define a range (i.e. state the lower value is 0).
Sec 4.1: "high density" -> "high mass"
It would help to have a diagram of \mathcal U, \mathcal S - \mathcal U, \alpha, \beta, \kappa.
It would be clearer if set notation was used for the complement of \mathcal U, rather than \beta's definition of s\notin \mathcal U.
Example 1: citation should be Ross 2010, not Ross 2011.
Example 1 has different notation than in Ross 2010 (consider changing to match)
It's possible that copying a model from the ensemble and fine-tuning it with the loss would yield a faster Algorithm (1). Would this work? What do the training curves (i.e. like the plots in Fig 3b) look like in that case?
Why does the breakout DRIL agent outperform the expert?
Mention that Pinkser's inequality yields the KL bound on total variation.


There is growing interest in designing lightweight neural networks for mobile and embedded vision applications. Previous works typically reduce computations from the structure level. For example, group convolution based methods reduce computations by factorizing a vanilla convolution into depth-wise and point-wise convolutions. Pruning based methods prune redundant connections in the network structure. In this paper, we explore the importance of network input for achieving optimal accuracy-efficiency trade-off. Reducing input scale is a simple yet effective way to reduce computational cost. It does not require careful network module design, specific hardware optimization and network retraining after pruning. Moreover, different input scales contain different representations to learn. We propose a framework to mutually learn from different input resolutions and network widths 1 . With the shared knowledge, our framework is able to find better width-resolution balance and capture multi-scale representations. It achieves consistently better ImageNet top-1 accuracy over US-Net (Yu & Huang, 2019) under different computation constraints, and outperforms the best compound scale model of EfficientNet (Tan & Le, 2019) by 1.5%. The superiority of our framework is also validated on COCO object detection and instance segmentation as well as transfer learning. Deep neural networks have triumphed over various perception tasks such as image classification (He et al., 2016;Huang et al., 2017b;Simonyan & Zisserman, 2014), object detection (Ren et al., 2015;Redmon et al., 2016) and semantic segmentation . However, deep networks usually require large computational resources, making them hard to deploy on mobile devices and embedded systems. This motivates research in reducing the redundancy in deep neural networks or designing light-weight structures. Specifically, MobileNet (Howard et al., 2017) factorizes a standard 3 × 3 convolution into a 3×3 depth-wise convolution and a 1×1 point-wise convolution. ShuffleNet (Zhang et al., 2018a) uses 1 × 1 group convolution to further reduce computations and proposes the shuffle operation to help information flow among different groups. Another kind of approach is to prune redundant connections in the networks. 1 Under review as a conference paper at ICLR 2020 in-network structure for multiple resources.  shares weights among different subnetworks and each sub-network has its own batch normalization layer. US-Net (Yu & Huang, 2019)   proposes to compute batch normalization statistics after training, and introduces two training techniques to train a network that is executable at any network widths. However, these works approach the problem only from network structure perspective, ignoring the importance of network input.

This paper proposes a multi-resolution training scheme for a slimmable network. The proposed method provides a new regime leveraging diverse image resolutions for training sub-networks, which enables efficient network scaling. Throughout the experiments, the authors claim the proposed method shows better performance compared to US-net.
Pros)
(+) The idea of multi-resolution training combining to a slimmable network looks good
(+) Applying a slimmable network's technique to other tasks including detection and segmentation looks good
(+) The combination of multi-resolution and the slimmable network seems to be reasonable.
(+) The paper is well written and looks justified well.
(+) The authors provided extensive experiments.
Cons)
(-) There is no backups why the proposed method could outperform over US-net.
(-) The proposed method is incremental and improvements are marginal.
(-) Looks like there exists missing in details of the experiments.
(-) The performance report of the compared methods is quite strange.
Comments)
- The proposed method is too straightforward, so the authors should clarify why it works over US-net. Additionally, can the authors provide advantages using a different image-scale need for training a different sub-network?
- The authors should clarify the training details of US-Net used in this paper. The performance of US-Net in Figure 4 (a) looks the same as the performance of US-Net trained with [0.05, 1]x scaling in the original paper. However, in the original paper, the authors of US-net reported [0.05, 1]x scaling as the worst performance setting in the original paper. Therefore, the authors should compare their method with the best performance setting of US-Net, which is [0.25, 1]x (because the proposed method looks being used [0.25, 1]x training setting, so the comparison should be done in fair).
- The scaling parameters of US-Net used in the experiments should be specified. All the results of US-Net do not contain where they come from (i.e., the training width bound in US-net).
- Can the authors report the results for 0.5-224 and 0.15-224 in Figure.4(a)? Why 0.7-160 and 0.25-160 were picked?
- In Table 1, the performance of EfficientNet is weird. EfficientNet-B2 has 79.8% accuracy with 1.0B FLOPs, but the reported performance in this paper of EfficientNet has 75.6% accuracy with 2.3B FLOPs. Please clarify this.
- How much does KLdiv contribute to the overall performance?
- All the tables are not clearly shown. Please reattach all the tables for better readability.
About rating)
I think the idea looks novel, but the method is quite straightforward, and the paper does not incorporate any analysis as a backup for the proposed method. The initial rating is towards reject, but I would like to see the authors' response and the other reviewers' comments. After that, the final rating might be changed.


We propose Zeno++, a new robust asynchronous Stochastic Gradient Descent (SGD) procedure which tolerates Byzantine failures of the workers. In contrast to previous work, Zeno++ removes some unrealistic restrictions on workerserver communications, allowing for fully asynchronous updates from anonymous workers, arbitrarily stale worker updates, and the possibility of an unbounded number of Byzantine workers. The key idea is to estimate the descent of the loss value after the candidate gradient is applied, where large descent values indicate that the update results in optimization progress. We prove the convergence of Zeno++ for non-convex problems under Byzantine failures. Experimental results show that Zeno++ outperforms existing approaches.

This paper proposes an approach to Byzantine fault tolerance in asynchronous distributed SGD. The approach appears to be novel. Theoretical convergence guarantees are provided, and an empirical evaluation illustrates very promising results.
Overall, the paper is well-written and the results appear to be novel and interesting. Although I have a few questions, listed below, I generally lean towards accepting this paper.
The assumption of a lower bound on validation gradients is somewhat troubling, especially for over-parameterized problems where so-called "interpolation" may be possible. I realize that validation samples are never used explicitly for stochastic gradient updates, but the algorithm does ensure that the stochastic gradients used are similar to gradients of validation samples. If one is converging to a (local) minimizer, one wants the gradient to vanish. How do we reconcile these points? Also, to properly set $\rho$, for the theory to be valid, one needs to know this bound (or a lower bound on $V_2$). Is this practical?
The paper claims that the computational overhead of Zeno+ is too great to evaluate for comparison with Zeno++. From my reading of the two methods, it isn't immediately obvious to me why this is the case. Including experiments which at least compare the per-iteration runtime (even if not running Zeno+ for training to completion) would make the paper more compelling. After all, Zeno++ still involves periodically evaluating the gradient at a validation sample.
The paper makes the reasonable point that it is not reasonable to assume a bounded number of adversaries in the asynchronous setting, and the theorem statements make no assumption about the number of adversaries or rate at which received gradients are from a Byzantine worker. However, there are also no guarantees about whether the algorithm will ever make progress (i.e., will line 8 ever be reached?). This should be stated more transparently in the paper. Also, I was wondering, given that a gradient has been computed on the parameter server's validation set, which is assumed to be "clean", why not take a step using this gradient when the test in line 7 fails?
Finally, the paper titles includes SGD, but the description in Def 1 doesn't appear to involve stochastic gradients. Typical parameter server implementations have workers compute mini-batch stochastic gradients, not full gradients on their shard of the training set. Does Zeno++ need to be modified to run in this setting? Does the theory still hold?
Minor:
- Is there a typo in line 5 of Zeno++? Should this be $\nabla f_s$ instead of $f_s$? Otherwise, what does it mean to take the inner product of $v$ and $g$ in line 7?


Models of code can learn distributed representations of a program's syntax and semantics to predict many non-trivial properties of a program. Recent state-ofthe-art models leverage highly structured representations of programs, such as trees, graphs and paths therein (e.g., data-flow relations), which are precise and abundantly available for code. This provides a strong inductive bias towards semantically meaningful relations, yielding more generalizable representations than classical sequence-based models. Unfortunately, these models primarily rely on graph-based message passing to represent relations in code, which makes them de facto local due to the high cost of message-passing steps, quite in contrast to modern, global sequence-based models, such as the Transformer. In this work, we bridge this divide between global and structured models by introducing two new hybrid model families that are both global and incorporate structural bias: Graph Sandwiches, which wrap traditional (gated) graph message-passing layers in sequential message-passing layers; and Graph Relational Embedding Attention Transformers (GREAT for short), which bias traditional Transformers with relational information from graph edge types. By studying a popular, non-trivial program repair task, variable-misuse identification, we explore the relative merits of traditional and hybrid model families for code representation. Starting with a graph-based model that already improves upon the prior state-of-the-art for this task by 20%, we show that our proposed hybrid models improve an additional 10-15%, while training both faster and using fewer parameters.

The paper proposes improvements on existing probabilistic models for code that predicts and repairs variable misuses. This is a variant of the task, proposed by Vasic et al. The task takes a dataset of python functions, introduces errors in these functions and makes a classifier that would identify what errors were introduced and effectively reconstruct the original code.
The paper claims to improve state-of-the-art results published by Vasic et al for this task, however the RNN model by Vasic et al was known to be far from optimal when the work was published. Furthermore, that task was evaluated on artificially introduced changes (the original code could contain an error), but it is not clear that the improvements would have any practical effect. In fact, I conjecture that the bug-detector is in fact worse, because the entire dataset is not sufficiently large for millions of parameters and it is not clear that bugs that were originally the dataset were not learned by the better model, making it worse at spotting them. Given the relatively thinner contribution on the rest of the paper, I think this would be a valid question to be addressed to show the effectiveness of the model beyond accuracy on the artificial task.
The paper does a number of contributions to the neural architecture. The most important change precision-wise is to use transformer model instead of RNN (the model used by Vasic et al).  This change is also what makes the work perform as well or better than GGNN-based approaches. The paper then proposes to improve the transformer model by modifying the attention where there are edges. The rest of the contributions seem to be addressing the problem of aster convergence speed.  The other contribution of the paper is by selecting which edges to include and it is also shown to improve convergence speed.
Given that most of the work talks about performance, it would also help if the authors clarify what kind of hardware was used and which optimizer.
Q: Why a larger transformer model was not evaluated?
More minor issues:
“We conjecture that the Transformer learns to infer many of the same connections”. There is no confirmation for this besides similar accuracy, but if this is the case, why would I change the architecture and not just try with initializing the vectors to values corresponding to this knowledge and get faster convergence?
page 3, “where q and k correspond to the query and key vectors as described above,”. It seems it is q_i and k_j?
Update after the rebuttal:
 - I thank the authors for running additional experiments on a short notice. I have some reservations about their correctness though (we still do not know if there were bugs fixed in these commits, their number is very low and the authors seem to have cherrypicked the numbers to show - their first updated revision had recall at 20%, then decided to show it at 10%, the RNN baseline is actually having the highest recall of all although it is not highlighted). I am not sure that data cleaning of this small evaluation sample will not show a different picture.
 - I actually increase my score a bit (I was torn in the beginning), because this is one of the first papers to run transformer model on code. I still think the actual contributions of the paper are minor.


A significant body of recent work has examined variational autoencoders as a powerful approach for tasks which involve modeling the distribution of complex data such as images and text. In this work, we present a framework for modeling multiple data sets which come from differing distributions but which share some common latent structure. By incorporating architectural constraints and using a mutual information regularized form of the variational objective, our method successfully models differing data populations while explicitly encouraging the isolation of the shared and private latent factors. This enables our model to learn useful shared structure across similar tasks and to disentangle cross-population representations in a weakly supervised way. We demonstrate the utility of our method on several applications including image denoising, sub-group discovery, and continual learning.

This paper introduces a novel model called "Cross-Population Variational Autoencoder (CPVAE), which is designed to model data from different distributions sharing some common structure. The proposed generative model utilizes both shared and private per-population latent variables. In order to restrict shared latent variables from "leaking" into private representations, the authors introduce an information-theoretic regularizer. This regularizer forces private population representations to: (a) maximize mutual information with input samples from their population, and (b) minimize mutual information with input samples from other populations. In other words, private representations are forced to be "meaningful" on the corresponding population alone.
Quality:
The paper is well-written. I find the proposed method to be quite interesting. The derivations appear to be correct except for possibly the cancellation in Eq. 5, which I originally missed.
Significance:
In my opinion, if sound, the approach discussed in this paper may lead to interesting practical applications and may inspire other methods based on similar ideas.
Originality:
Even though, as authors point out, there is a substantial amount of work in this field, I believe that their approach is novel and has its own merits.
Clarity:
The paper is well written and the material is presented with clarity. In my opinion, the only exception is Section 4.4, which could definitely benefit from a few additional sentences describing the training procedure in more detail. Right now I find it a bit confusing. It would appear that the shared encoder / decoder continue to be trained as new populations arrive. Would this mean that catastrophic forgetting can actually impact this shared representation? And if it changes by a sufficient degree, can it reduce the quality of the generated samples for older populations? If so, I think these points should be mentioned in the text.
Questions and suggestions:
Experiments described in the paper are sufficiently convincing, but there are a few questions that could potentially be better clarified in the paper.
1. After reviewing the text again and seeing the comment of Reviewer #1, I am also confused about the cancellation in  I_q(x_k; t_k) - I_q(x_{-k}; \tilde{t}_k). Is it not true that marginal distributions q_\phi(t) and q_\phi(\tilde{t}) in the KL divergence term in Eq. 5 are different? Unfortunately, the final optimization objective relies on the cancellation of these terms and if they do not cancel, the approach may not be theoretically justified despite producing interesting and compelling results. (This affected the final rating. I will be able to change the rating once this point is clarified.)
2. Another issue is related to the special case when there are several very similar populations. Consider, for example, a case when there are two nearly-identical populations out of many. Using very similar latent variables for two similar populations would be penalized by the regularizer (not too significantly though). I assume that depending on the embedding sizes and the value of alpha (which authors introduce in Section 2.3), the model would either choose to use shared latent variables to encode these populations, or would allow for two nearly-identical private latent representations to exist. I think this is a conceptually important special case that could be mentioned and possibly explained in the paper.
3. I think the paper would also benefit from a clarification regarding the "mixing" function g. Choosing this function to be a simple sum of arguments seems restricting and may be insufficient for some datasets. It does not appear to be the case, but are there any restrictions on g? Can it come from a parametrized function family with parameters being optimized during training?
4. I think the paper would benefit from a more detailed discussion in Section 4.4 (see above).


Self-explaining models are models that reveal decision making parameters in an interpretable manner so that the model reasoning process can be directly understood by human beings. General Linear Models (GLMs) are self-explaining because the model weights directly show how each feature contributes to the output value. However, deep neural networks (DNNs) are in general not self-explaining due to the non-linearity of the activation functions, complex architectures, obscure feature extraction and transformation process. In this work, we illustrate the fact that existing deep architectures are hard to interpret because each hidden layer carries a mix of low level features and high level features. As a solution, we propose a novel feature leveling architecture that isolates low level features from high level features on a per-layer basis to better utilize the GLM layer in the proposed architecture for interpretation. Experimental results show that our modified models are able to achieve competitive results comparing to main-stream architectures on standard datasets while being more self-explainable. Our implementations and configurations are publicly available for reproductions †.

This paper proposes a feature leveling technique to improve the self-explaining of deep fully connected neural networks. The authors propose to learn a gated function for each feature dimension for whether to directly send the feature to the final linear layer. The gated function is trained with L0 regularization technique proposed in (Louizos et al., 2017) to encourage more low-level features passed to the final layer. Experimental results on MNIST, California Housing, CIFAR10 show that the proposed method can achieve comparable performance with existing algorithms on sparse neural network training.
Quality:
Overall, the paper is well written with some minor formatting errors. The toy example demonstrates the idea of this paper clearly. However, the novelty of this paper, when compared to NIT, is that the L0 regularization is used to pass the feature to last layer. Considering the self-explaining feature, this work can only explain that some of the input features are suited for final layer, while there is no explanation on the other features since they are used to construct higher level features.
Claririty:
Some parts of this paper are not clear:
1.	Why l_k and h_k has to be disjoint? A feature suited for final classification does not suggest that it can’t be used to construct higher-level feature.
2.	In (4), should not B() be an inverse binary activation function: (1-z)?
3.	Is g(.) the Bernoulli distribution?
4.	In section 5.2, why compare the gradients of a specific input example while one can directly look at z_k, the gated function?
5.	I assume the fully connected layers have bias term. If so, (4) suggests that the gated location will also be added with a learned bias, which is different than what the paper proposes.
Novelty:
The novelty of this paper lies in the sparse training objective becomes passing as many lower-level features to final layer as possible instead of zeros out the intermediate weights. However, the key technique, L0 regularization, has been proposed and used as stated in the related work. While the authors state the application L0 to a novel context to select features is different from prior work, the novelty is rather incremental.
Significance:
This work demonstrates that the L0 regularization technique for sparse neural network training can also be applied to learn a skip-layer connection. However, from both novelty, performance, and self-explaining perspectives, this work does not introduce much to the field.
Pros:
1.	The paper is well written.
2.	The toy example showcases the issue that this work tries to tackle.
3.	The experimental results show the comparable performance to existing works.
Cons:
1.	The novelty is not sufficient considering the prior works on sparse neural network training.
2.	There are some clarification issues as mentioned before.
3.	The performance is only comparable to existing works.
4.	The self-explaining contribution is not clear since only a few input features can be explained if they are passed to the final layers.
5.	There is no experiment on how \lambda would affect the resulting network architecture.
Minor corrections:
1.	First paragraph on sec. 5: three datasets: two datasets (or mention it’s in appendix).
2.	5.2 compare to NIT: the citations are in wrong format. Also the reference for NIT is corrupted.


We present Multitask Soft Option Learning (MSOL), a hierarchical multitask framework based on Planning as Inference. MSOL extends the concept of options, using separate variational posteriors for each task, regularized by a shared prior. This allows fine-tuning of options for new tasks without forgetting their learned policies, leading to faster training without reducing the expressiveness of the hierarchical policy. MSOL avoids several instabilities during training in a multitask setting and provides a natural way to learn both intra-option policies and their terminations. We demonstrate empirically that MSOL significantly outperforms both hierarchical and flat transfer-learning baselines in challenging multi-task environments.

Summary:
The authors propose a method for learning hierarchical policies in a multi-task setting built on options and casts the concepts into the Planning as Inference framework. They claim that the method brings several advantages such as:
-Stabilize end-to-end mtl,
-Leads to coordination b/w master policies,
-Allows fine-tuning of options
-Learn termination policies naturally
The proposed approach derives the reward Eq.6 by extending the graph to options for Levine’s tutorial. Eq.6 is simply the extension of the reward of maximum entropy RL to the options framework. The ideas presented in the paper are interesting, but I have concerns about the scalability of such an approach. Please see the detailed comments below.  Additionally, please note that although I have marked a weak reject, I am open to adjusting my score if the rebuttal addresses enough issues.
Detailed Comments:
A primary weakness of this approach is that it seems like there is one network that learns the options and is shared across all task (that would be the prior) and then there is a task-specific network for all options (posterior), wouldn’t this be very difficult to scale if we want to learn reusable options over the lifetime of an agent? If there are n tasks, do you need to use n different networks?
The authors assume that all options are present everywhere i.e. I ⊆ S. I think the work could benefit from removing this assumption.
The authors mention that unlike (Frans et al., 2018), they learn both intra-option and termination policies: there is definitely more work that aims to learn both the skill and its termination. It would be more complete to cite additional references here that learn both of these or rephrase this sentence.
It does not seem clear why “term 1 of the regularization is only nonzero whenever we terminate an option and the master policy needs to select another to execute.” This doesn’t seem true as this is a ratio of the two probabilities and not just the instantiation of the random variable.
The results in moving bandits alone are very convincing. However, in Taxi (2b) distral+action seems to be as good as/even better MSOL. In directional Taxi (2c) Distral(+action) manages to reach the same final performance (if we care about that), can you please comment on this.
Some parts of the experiments section does not seem clear to me, Does the proposed approach use a network per task? if yes, then it is obvious that their method could improve over learning on 12 tasks with one set of network. Please clarify.
One major concern is that the only high dimensional experiment is a swimmer and it is not immediately clear how much do we gain there. Distral is relatively closer in performance to both MSOL and MSOL frozen. I would recommend evaluation in a variety of high-dimensional domains such as other instances in Mujoco, and visual domains. In particular, the proposed ideas would make a stronger case if the baselines included other multitask hierarchical agents such as [4] for example. A discussion including some of the missing relevant related multi-task literature would also be helpful [1,2,4,5,6].
[1] Mann, T. A., Mannor, S., & Precup, D. (2015). Approximate value iteration with temporally extended actions. JAIR, 53, 375-438.
[2] Konidaris, G., & Barto, A. G. (2007). Building Portable Options: Skill Transfer in Reinforcement Learning. In IJCAI,
[3] Andreas, J., Klein, D., & Levine, S. (2017). Modular multitask reinforcement learning with policy sketches. ICML
[4] Tessler, Chen, et al. "A deep hierarchical approach to lifelong learning in minecraft." Thirty-First AAAI Conference on Artificial Intelligence. 2017.
[5] Ammar, Haitham Bou, et al. "Online multi-task learning for policy gradient methods." International Conference on Machine Learning. 2014.
[6] Mankowitz, Daniel J., Timothy A. Mann, and Shie Mannor. "Adaptive skills adaptive partitions (ASAP)." Advances in Neural Information Processing Systems. 2016.


Achieving fusion of deep learning with combinatorial algorithms promises transformative changes to artificial intelligence. One possible approach is to introduce combinatorial building blocks into neural networks. Such end-to-end architectures have the potential to tackle combinatorial problems on raw input data such as ensuring global consistency in multi-object tracking or route planning on maps in robotics. In this work, we present a method that implements an efficient backward pass through blackbox implementations of combinatorial solvers with linear objective functions. We provide both theoretical and experimental backing. In particular, we incorporate the Gurobi MIP solver, Blossom V algorithm, and Dijkstra's algorithm into architectures that extract suitable features from raw inputs for the traveling salesman problem, the min-cost perfect matching problem and the shortest path problem.

=== Summary ===
The authors propose a method for efficiently backpropagating through unmodified blackbox implementations of exact combinatorial solvers with linear objective functions.
The gradient of such exact combinatorial solvers exists almost everywhere but is zero. The authors remark that the loss has the same gradient wrt to the solver's input as its linearization around the solver's input.  They therefore propose to interpolate the loss' linearization with a continuous (piecewise affine) function and use the gradient of this interpolation to backpropagate through the solver. This gradient is obtained efficiently by simply calling the solver on a single perturbed input (the perturbation depends on the incoming gradient, ie the gradient of the loss wrt to the solver's output).
The authors further study the properties of this piecewise affine interpolation and characterize its interpolation behavior as a function of a hyperparameter which controls the trade-off between "how informative the gradient is" and "how faithful the interpolation is to the original solver".
The authors validate their method with experiments on synthetic tasks that have both a visual processing aspect and a combinatorial aspect:
    - Shortest Path on Warcraft II terrain maps
    - TSP between country capitals where the inputs to the convnet are country flags
    - Min-cost perfect matching from Mnist digits.
Specifically, they feed the output of a convnet to the relevant solver (depending on the task) and learn end-to-end by backpropagating through the solver with their proposed method. They show that their method successfully solves the tasks where baseline ConvNet architectures fail.
=== Recommendation ===
This paper addresses an important problem and presents a novel approach.
Methods for combining combinatorial optimization algorithms and machine learning usually rely on modifiying or relaxing the combinatorial problem itself which prevents using solvers as-is.
In contrast, the presented method allows to efficiently backpropagate through unmodified implementations of blackbox exact solvers with a linear objective. AFAIK this is the first method that allows this.
A weakness of the paper is that the experiments only validate proof of concept (as noted by the authors). They are small-scale and only compare against conventional ConvNets baselines (as opposed to other approaches to backpropagate through relaxed combinatorial problems).
Additionally, the characterization of the interpolation (whose gradient is used) doesn't directly explain why the gradient of the interpolation is a reasonable choice.
Overall, I recommend for acceptance.
=== Questions / Comments ===
- The authors show properties related to the interpolation behavior of the proposed interpolation function. What is the actual point/benefit of satisfying these properties? Are there arguments for why this is important besides the experimental results?  Is the point that since lambda controls how "faithful vs informative" the gradient is , there must be a range of values for lambda for which the method works?
- It would be interesting to have experiments with non-exact solvers
- It would be interesting to optimize directly for the combinatorial objective in the experiments (using a policy gradient for example) rather than perform supervised learning on the solutions.
- Consider adding related work subsection on argmin optimization and meta-learning.


Evolution strategies or zeroth-order optimization algorithms have become popular in some areas of optimization and machine learning where only the oracle of function value evaluations is available. The central idea in the design of the algorithms is by querying function values of some perturbed points in the neighborhood of the current update and constructing a pseudo-gradient using the function values. In recent years, there is a growing interest in developing new ways of perturbation. Though the new perturbation methods are well motivating, most of them are criticized for lack of convergence guarantees even when the underlying function is convex. Perhaps the only methods that enjoy convergence guarantees are the ones that sample the perturbed points uniformly from a unit sphere or from a multivariate Gaussian distribution with an isotropic covariance. In this work, we tackle the non-convergence issue and propose sampling perturbed points from a mixture of distributions. Experiments show that our proposed method can identify the best perturbation scheme for the convergence and might also help to leverage the complementariness of different perturbation schemes.

This paper proposes to change the sampling scheme of evolutionary strategies to sample from a mixture of distributions (instead of the standard way of sampling from one Gaussian).
Although this seems to provide slightly better performance, the contribution is incremental and more importantly the paper lacks clarity, especially regarding the theory part. The experiments are also not on par with what’s typically expected in the literature. Overall, this paper is not ready for acceptance at a venue such as ICLR.
EXP3.P algorithm
This is one of the key components of the algorithm presented by the authors but it is not properly explained in the paper. In fact, the algorithm implemented by the authors is very unclear to me. Below I quote the description given by the authors:
“One can view that EXP3.P runs synchronously with Algorithm 5 in the way that step 4 of EXP3.P implements step 5 of Algorithm 5 and that step 5 and the subsequent steps of EXP3.P are conducted after step 8 of Algorithm 5 is finished“
This is a very poor description of the algorithm. The authors should give a clear pseudo-code to explain the algorithm.
Theoretical guarantees
1) Again, the paper is very unclear on what algorithm is being implemented. The authors claim they can directly use the guarantees provided by Bubeck et al but this is unclear to me given that the algorithm is never fully stated. This needs to be clarified.
2) The authors claim asymptotic convergence, specifically they claim “Therefore, one might show that the proposed algorithm converges with a high probability.“. This proof should be provided in the paper, not left as an exercise to the reader...
3) Does Theorem 2 (Theorem 3.3 from Bubeck) require convexity of the function? Can you comment on guarantees for non-convex functions?
Prior work
The authors do not give a clear description of prior work. There is a large literature on proving convergence guarantees under weaker assumptions than discussed in the paper, see e.g.
Y. Diouane, S. Gratton, and L. N. Vicente. Globally convergent evolution strategies. Math.
Program., 152:467–490, 2015.
Y. Diouane, S. Gratton, and L. N. Vicente. Globally convergent evolution strategies for
constrained optimization. Comput. Optim. Appl., 62:323–346, 2015.
Vincente. Worst case complexity of direct search
Experiments
This is also a weak part of the paper.
1) The authors only run experiments on very small datasets. Regarding the non-convex task, it would be much more interesting to consider a reinforcement learning task where evolutionary methods typically perform well.
2) You should include stronger baselines too, e.g. https://arxiv.org/abs/1806.10230.
3) An interesting question to look at would be to analyze the behavior of the algorithm as a function of the number of mixture components. Also, what is the computational increase in terms of the number of components? Consider giving plots in terms of computation time.


Many irregular domains such as social networks, financial transactions, neuron connections, and natural language structures are represented as graphs. In recent years, a variety of graph neural networks (GNNs) have been successfully applied for representation learning and prediction on such graphs. However, in many of the applications, the underlying graph changes over time and existing GNNs are inadequate for handling such dynamic graphs. In this paper we propose a novel technique for learning embeddings of dynamic graphs based on a tensor algebra framework. Our method extends the popular graph convolutional network (GCN) for learning representations of dynamic graphs using the recently proposed tensor M-product technique. Theoretical results that establish the connection between the proposed tensor approach and spectral convolution of tensors are developed. Numerical experiments on real datasets demonstrate the usefulness of the proposed method for an edge classification task on dynamic graphs.

This paper presents a M-product based temporal GCNs to handle dynamic graphs. Experiments on four real datasets are performed to verify the effectiveness of the proposed model.
Overall, I think this paper make a few contributions to advocate tensor M-product. However, there are several big issues as listed below. Given the current status, I could not accept the paper.
Pros:
1, The generalization brought by M-product seems to be general as it includes quite a few graph convolution elements for 3D tensors in a natural way.
2, The experimental setup is reasonable. Datasets are collected from practical problems and of moderately large scale.
3, The paper is clearly written and easy to follow.
Cons & Questions:
1, My first concern is that M-product formulation does not bring any new insights as people have already used some of the key elements in practice for a long time. For example, the M-transform is just applying 1x1 convolution to multi-channel image. Slice-wise matrix multiplication is also common in practice.
2, Moreover, I think there are several challenges in the M-product formulation which prevent the technique from being practical.
(1) Sharing M such that frontal slices of the transformed signal are the same, i.e., each row of M share the same vector, limits the model capacity significantly. If there is no sharing mechanism, then the model learned on one sequence of graphs could not be applied to another sequence of graphs given two sequences have different lengths.
(2) If you learn M from data, how could you ensure that M is invertible? In the paragraph before section 4.1, an edge classification formulation is proposed where the inverse M-transform is abandoned. However, if in practice, you do not need the inverse transform, then do those theoretical properties still hold and what is the meaning of introducing such M-product formulation?
3, A few temporal GCN baselines are neither compared or discussed, e.g., [1].
4, Could you explain why all the other GCN variants performs significantly worse with a symmetrized adjacency matrix compared to using the asymmetric one?
[1] Li, Y., Yu, R., Shahabi, C. and Liu, Y., 2017. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. arXiv preprint arXiv:1707.01926.
======================================================================================================
After I read authors' reply and other reviewers' comments, I would like to keep my original rating as the issues have not been properly addressed. I agree with the Reviewer #4 that the theoretical results are a bit artificial and trivial.


We show state-of-the-art word representation learning methods maximize an objective function that is a lower bound on the mutual information between different parts of a word sequence (i.e., a sentence). Our formulation provides an alternative perspective that unifies classical word embedding models (e.g., Skip-gram) and modern contextual embeddings (e.g., BERT, XLNet). In addition to enhancing our theoretical understanding of these methods, our derivation leads to a principled framework that can be used to construct new self-supervised tasks. We provide an example by drawing inspirations from related methods based on mutual information maximization that have been successful in computer vision, and introduce a simple self-supervised objective that maximizes the mutual information between a global sentence representation and n-grams in the sentence. Our analysis offers a holistic view of representation learning methods to transfer knowledge and translate progress across multiple domains (e.g., natural language processing, computer vision, audio processing).

The paper proposes to make a clear connection between the InfoNCE learning objective (which is a lower bound of the mutual information) and multiple language models like BERT and XLN. Then based on the observation that classical LM can be seen as instances of InfoNCE, they propose a new (InfoWord) model relying on the same principles, but taking inspiration from other models also based on InfoNCE. Mainly, the proposed model  differs both in the nature of the a and b variables used in InfoNCE, and also on the fact that it uses negative sampling instead of softmax. Experiments are made on two tasks and compared to a classical BERT model, and on the BERT-NCE model that is a BERT variant proposed by the authors which is somehow in-between BERT and InfoWord. They show that their approach works quite well.
I have a very mitigated opinion on the paper. I) First, I really like the idea of trying to unify different models under the same learning principles, and then show that these models can be seen as specific instances of generic principles. But the way it is presented and explained lacks of clarity: for instance in Section 2, some notations are not well defined (e.g what is f?) . Moreover, the way classical models are casted under the InfoNCE principle is badly written: it assumes that readers have a very good knowledge of the models, and the paper does not show well the mapping between the loss function of each model and the InfoNCE criterion. It gives technical details that could (in my opinion) get ignored, and I would clearly prefer to catch the main differences between the different models that being flooded by technical details. So, my suggestion would be to improve the writing of this section to make the message stronger and relevant for a larger audience. II) The Infoword model can be seen as a simple instance of word masking based models, and as an extension of deep infomax for sequences (it would be certainly nice to describe a little bit what Deep InfoMax is to facilitate the reading).  Here again, the article moves from technical details (e.g "hidden state of the first token (assumed to be a special start of sentence symbol ") without providing formal definitions. Having a first loss function after paragraph 4 could help to understand the principle of this model (before restricting the model to n-grams).  Moreover, the equation J_DIM seems to be wrong since it contains g_\omega twice while I think (but maybe I am wrong) that it has also to be defined by g_\psi. J_MLM is also not clear since x_i is never defined (I assume it is x_{i:i}). At last,  after unifying multiple models under one common learning objective, the authors propose to mix two different losses which is strange (the effect of the second term is slightly studied in the experimental section) without allowing us to understand why it is important to have this second loss function and why the first one is not sufficient enough. At last, I am pretty sure to not be able to reproduce the model described in the paper (adding a section on that in the supplementary material would help), and many concrete aspects are described too fast (like the way to sample negative pairs).
Concerning the experimental section, experiments are convincing and show that the model is able to achieve a performance which is close to classical models. In my opinion, tis section has to be interpreted as  a proof that the proposed unified vision is a good way to easily define new and efficient models.
To summarize, the unification under the InfoNCE principle is interesting,  but the way the paper is written makes it very difficult to follow, and the description of the proposed model is unclear (making the experiments difficult to reproduce) and lacks of a better discussion about the interest of mixing multiple loss.


We design and conduct a simple experiment to study whether neural networks can perform several steps of approximate reasoning in a fixed dimensional latent space. The set of rewrites (i.e. transformations) that can be successfully performed on a statement represents essential semantic features of the statement. We can compress this information by embedding the formula in a vector space, such that the vector associated with a statement can be used to predict whether a statement can be rewritten by other theorems. Predicting the embedding of a formula generated by some rewrite rule is naturally viewed as approximate reasoning in the latent space. In order to measure the effectiveness of this reasoning, we perform approximate deduction sequences in the latent space and use the resulting embedding to inform the semantic features of the corresponding formal statement (which is obtained by performing the corresponding rewrite sequence using real formulas). Our experiments show that graph neural networks can make non-trivial predictions about the rewrite-success of statements, even when they propagate predicted latent representations for several steps. Since our corpus of mathematical formulas includes a wide variety of mathematical disciplines, this experiment is a strong indicator for the feasibility of deduction in latent space in general.

The paper proposes a method to do math reasoning purely using formula embeddings. The proposed method employs a graph neural network to embed math formulas to a latent space. The formula embeddings are then combined with theorem embeddings (also formulas, computed in the same way as formula embeddings) to predict whether one can do one step of math reasoning using the corresponding theorem, and also to predict the embeddings of the resulting formula. Empirically the authors demonstrate that the method can be chained end-to-end to do multiple steps of reasoning purely in the latent space.
I tend to accept this paper, (but also OK if it gets rejected), for the following reasons: (1) the idea is novel and interesting; (2) the writing of the paper is below conference standard and very hard to read, especially the method and the experiment sections.
===========================================================================
Novelty and significance
I really like the idea of doing math reasoning in latent space. The idea is definitely novel and interesting. It is related to existing works such as neural logic induction[1] and planning in latent space[2]. It is amazing that one can do multiple steps of math reasoning after only training the model using data from one single step. It would be interesting to see how it can improve existing learning-based theorem provers.
My question is if we want to integrate the proposed method into theorem provers, after multiple steps of math reasoning, how would us know the goal has been proved? Is it possible that we can train a decoder that maps back from the latent space to the formula space? Also can it work with theorems that decompose the current goal into several sub-goals? I know these are not the concerns of this paper, but I would be really grateful if you could provide some intuitive answers!
===========================================================================
Writing
The paper is not well-organized and not written in a consistent way. For the method and the experiment sections, I need to jump back and forth several times in order to understand what the authors are trying to say.
1. Typo: Third paragraph in section 1, "...which is makes use of ...".
2. It's very confusing when the authors introduce \sigma and \omega in the beginning of section 4: why would you need two networks predict the same thing?
3. Mentioning "merging \sigma and \omega, is left for future work" is confusing before formally introducing \sigma and \omega.
4. Even when the authors formally introduce \sigma and \omega in 4.2, it is still not clear that why both of them are used for modelling the success probability.
5. In fact, I don't know why \omega needs to output p. It's never mentioned in the experiment section.
6. The rationale of the two tower design (why not combine two) is not clearly explained.
7. Typo: Page 5 last paragraph, "... negative instances for for each ...".
8. The itemized part in 5.3, "...carefully selected baselines: 1.xxx, 2.xxx, 3. xxx, 4. xxx". However, both 3 and 4 are not baselines!
9. It is not clear that baseline 1 and 2 correspond to which baselines in later experiments.
10. Reading the baselines before the experiments is very confusing. For example, for baseline 1, it is very hard to understand why would we want to use such an unusual baseline, and why it is called a "random baseline".
11. Baseline 2 is actually referred to as "usage baseline" but this name is not introduced in the itemized part.
[1] Rocktäschel, Tim, and Sebastian Riedel. "End-to-end differentiable proving." Advances in Neural Information Processing Systems. 2017.
[2] Srinivas, Aravind, et al. "Universal planning networks." arXiv preprint arXiv:1804.00645 (2018).


Imitation Learning (IL) is an appealing approach to learn desirable autonomous behavior. However, directing IL to achieve arbitrary goals is difficult. In contrast, planning-based algorithms use dynamics models and reward functions to achieve goals. Yet, reward functions that evoke desirable behavior are often difficult to specify. In this paper, we propose "Imitative Models" to combine the benefits of IL and goal-directed planning. Imitative Models are probabilistic predictive models of desirable behavior able to plan interpretable expert-like trajectories to achieve specified goals. We derive families of flexible goal objectives, including constrained goal regions, unconstrained goal sets, and energy-based goals. We show that our method can use these objectives to successfully direct behavior. Our method substantially outperforms six IL approaches and a planning-based approach in a dynamic simulated autonomous driving task, and is efficiently learned from expert demonstrations without online data collection. We also show our approach is robust to poorly specified goals, such as goals on the wrong side of the road.

Summary:
- key problem: expert-like probabilistic online motion planning to reach arbitrary goals without reward shaping thanks to off-line learning from expert demonstrations;
- contributions: 1) an imitative planning procedure via gradient-based log-likelihood maximization leveraging "imitative models" q(future states | features), 2) multiple proposals to define flexible goals in this probabilistic framework, 3) a complete implementation for end-to-end navigation in CARLA, 4) an extensive experimental evaluation showcasing the performance, flexibility, interpretability, and robustness of the proposed approach w.r.t. the previous state of the art and several Imitation Learning (IL) and Model-Based Reinforcement Learning (MBRL) baselines.
Recommendation: weak accept (leaning towards strong accept)
Key reason 1: principled probabilistic framework bringing the best of both IL and MBRL worlds.
- this planning as inference method is very succinctly and elegantly described in the paper with enough details in appendix (+ code) to suggest a high chance of reproducibility;
- the flexibility of defining different interpretable goals (6 different types explored in the paper) highlights the versatility of the approach;
- the additional benefits in terms of plan reliability estimation (Appendix E) are significant;
- the paper showcases how powerful and useful a good "imitative model" can be, therefore, reinforcing the interest of the research community in the important topic of off-line learning from large datasets of demonstrations (without requiring costly on-line data collection).
Key reason 2: thorough experimental evaluation with convincing results.
- the experimental protocol used is the standard one on CARLA and the results are state of the art;
- the comparison with related works, including recent ones, is thorough and well explained;
- the additional claims regarding robustness are substantiated by multiple experiments.
Suggested improvements:
- Not needing reward engineering is a major claim of this approach, but it seems that constructing goal likelihoods could be seen as a form of reward engineering, no? Table 3 indeed reports significant performance differences (absolute and relative) depending on how the goals are specified, especially in dynamic environments. As the experiments aim at maximizing the same performance metrics, is there a preferred goal type that works well across all experiments? If not, how is "goal definition" different than "reward engineering"?
- Could the authors please include a variance analysis (using different seeds) in Tables 3 and 4? Previous papers have reported high variance in similar settings (cf., Codevilla et al 2019), and this is a common issue in IL/RL.
- How important is knowing \lambda (traffic light state) perfectly in practice? Can the robustness to noise in \lambda be experimentally assessed? I would also clarify in section 4 and Table 2 that other methods do not use \lambda (the traffic light state), which is a signal very strongly correlated with the "ran red light" metric.
- More generally, what is the robustness of this approach to uncertainty / noise in \phi? Although it is typically available (as the authors mentioned) it is never perfect in practice. Can this be handled in a principled probabilistic way as an extension of the current formulation?
- The current model does not factor the influence of the agent on its environment (\phi := \phi_{t=0}). Is this framework limited to open loop planning, or does this open interesting future research directions towards closing the loop? It seems to be a key open problem to at least discuss in Section 5.
Additional Feedback:
- Figure 5 is confusing, not sure it adds much value to the paper;
- typos in Appendix ("pesudocode", "baselines that predicts", "search search").
 ## Update post rebuttal
Thanks to the authors' excellent replies and my initial inclination towards strong accept, I am happy to bump my score to 8. The authors did an excellent job, their rebuttal is on point, not avoiding hard questions, running additional requested experiments (incl. in a clever way for the most computationally expensive ones), and showing clear insights in future steps. Great job!


In some important computer vision domains, such as medical or hyperspectral imaging, we care about the classification of tiny objects in large images. However, most Convolutional Neural Networks (CNNs) for image classification were developed using biased datasets that contain large objects, in mostly central image positions. To assess whether classical CNN architectures work well for tiny object classification we build a comprehensive testbed containing two datasets: one derived from MNIST digits and one from histopathology images. This testbed allows controlled experiments to stress-test CNN architectures with a broad spectrum of signal-to-noise ratios. Our observations indicate that: (1) There exists a limit to signal-to-noise below which CNNs fail to generalize and that this limit is affected by dataset size -more data leading to better performances; however, the amount of training data required for the model to generalize scales rapidly with the inverse of the object-to-image ratio (2) in general, higher capacity models exhibit better generalization; (3) when knowing the approximate object sizes, adapting receptive field is beneficial; and (4) for very small signal-to-noise ratio the choice of global pooling operation affects optimization, whereas for relatively large signal-to-noise values, all tested global pooling operations exhibit similar performance.

The submission proposes an analysis of the impact of object size in images when performing classification tasks using neural networks of the BagNet family. The analysis is performed on two datasets, a large resolution cluttered MNIST and a histopathology dataset named nCAMELYON.
The paper attack interesting questions and links the size of the object in the image (O2I) to the training dataset size required. Also, showing that max-pooling is the only pooling operation that converges for very low O2I (but is the slowest to converge at higher O2I) is interesting and encourages discussion about the training (optimization) process.
In my opinion, the main issue about the submission is the limited depth in the contributions, analyzing a single family of network architectures (BagNets) over two datasets, one of which is relatively small. The family of R-CNN and its derivatives were especially designed to counter the impact of object size, it would have been interesting to include them in the analysis. Furthermore, limited insights can be carried out for tasks related to classification such as localization and segmentation. Models such as Single Shot Detectors split the image into grids and variable anchor sizes to perform their inference, are they affected to a lesser extent by object size? The limited insights provided to potential future readers prevents me from recommending the submission for acceptance.
On p. 7 (and fig. 7 (b) ), it is said that the lower performance of the larger receptive field suggests that class-relevant information is contained in the texture. I am not sure about this remark, I would have expected the network to learn to focus on the right regions, provided the receptive field is big enough to see the whole object of interest. Could the decrease in performance be attributed to the increased amount of learnable parameters that ended up too large for the “relatively small nCAMELYON dataset used for training”? (sec. 3.1, Global pooling operations)
Fig. 16 seems to suggest that multiple numbers can overlap significantly in nMNIST, yielding potentially confusing images even for humans. I am not sure if this is a desirable characteristic for such dataset.
Minor details
- Please use \cdot instead of the asterisk operator to denote multiplication (sec. 3, footnote 3, fig. 5 (b-c) and sec. 3.1;
- Fig. 4 uses two different (and non-linear) values for the x-axes, giving the impression that both curves can be compared, that might be confusing to the reader;
- Sec 1 and 5 “pixel-level level annotations”: duplicate “level”;
- p. 6 “1 and , 2”: extraneous comma.


In this work we delve deep in the Transformer architecture by investigating two of its core components: self-attention and contextual embeddings. In particular, we study the identifiability of attention weights and token embeddings, and the aggregation of context into hidden tokens. We show that attention weights are not unique and propose effective attention as a complementary tool for improving explanatory interpretations based on attention. Furthermore, we are the first to show that input tokens retain to a large degree their identity across the model. We also study how identity information propagates and find that it is mainly encoded in the angle of the embeddings and gradually decreases with depth. Finally, we demonstrate strong mixing of input information in the generation of contextual embeddings by means of a novel quantification method based on gradient attribution. Overall, we show that self-attention distributions are not directly interpretable and present tools to further investigate Transformer models.

CONTRIBUTIONS:
Topic: Analysis of self-attention in the standard Transformer, applied to models of language.
Notation for review (see also definitions in C1-C4): cell(t,L) is the cell in layer L at input-string-position t (“corresponding to input token t”); Z(t,L) is the output vector for cell(t,L). Also, in the review, points are sometimes labeled by “(label)”, which is intended to label the point that immediately *follows* it in the text.
C1. “Attention identifiability.” Uniquely identifying the attention-weight vector from the output of attention is not possible when D < T: For an individual head, the vector R returned by attention uniquely determines the attention-weight vector A only if the dimension D of the value vector equals or exceeds the length T of the input.
C2. “Effective attention vector” A1: When D < T, there is a method for uniquely decomposing A as A = A1 + A2 where A2 has no effect on R.
C3. “Token identifiability” method M. Identifying t from Z(t,L): A method M is proposed for this, and experimental results presented showing that, roughly, M succeeds at lower levels L but can fail at higher levels.
C4. “Hidden token attribution” method: A gradient-attribution-based method is proposed to quantify C(t,t’), the proportional contribution of input token t’ to Z(t,L). C(t,t) is the “[same-]token contribution”; the “contextual contributions” are the C(t,t’) for t’ =! t. Experimental results are presented showing that after L = 1, 6, 12 the median token values are about 30%, 15%, 10%.
MAIN ARGUMENT: Interpretation of the weight of cell(t,L)’s attention to cell(t’,L) as a measure of the influence of token t’ on the level-L representation of token t is (P0) frequent, but problematic, because (P1) the attention weight itself includes an irrelevant quantity (the A2 contribution, which should be omitted to get the effective attention [C2]) and because (P2) Z(t,L) is not properly viewed as representing token t, since (P2) it can be insufficient to even determine which token t it corresponds to [C3] as (P3) it can contain strong contributions from other tokens [C4].
RATING: Weak accept
REASONS FOR RATING (SUMMARY). The paper raises important challenges to a standard practice for interpreting the knowledge in Transformer models (especially BERT) by examining attention weights. It proposes methods for reaching more theoretically justified interpretations. It introduces important new general interpretive concepts in the process. The main argument is sufficiently well supported by theoretical and empirical results to justify making the ICLR community aware of it, although there is a missing step in the argument: justification that the proposed means of resolving the unidentifiability of attention yields an optimal result.
REVIEW
Weaknesses.
Minor complaint concerning exposition: The central result, C1 above, is actually rather obvious, unless I’m missing a subtlety (in which case it would be useful for that subtlety to be pointed out explicitly in the paper). (*) The dimension-D vector R is a linear combination of T vectors; if T > D, these T vectors cannot be linearly independent, so the weighting coefficients that yield R cannot be unique. The argument presented is rather overblown and may hide the obvious correctness of the result. However, I think over-deriving is clearly preferable to under-deriving, so this complaint truly is extremely minor. It might be worthwhile to add a sentence like (*) however to make sure that simple observation doesn’t get lost in the formalism. A further advantage of the more detailed derivation is that it sets up nicely the unique decomposition of A into the component A2 in the null space and the component A1 orthogonal to the null space (of the matrix containing the T vectors being linearly combined).
A more substantial question here: What argument or evidence is there that this particular decomposition A = A1 + A2  is the one that yields a definition of “effective attention” that is best for interpretative purposes? Is there a metric of success that would validate this? Given that A is underdetermined, there is a whole subspace of attention-weight vectors that yield the same result R of attention. The proposed choice of A1 within that space is a natural one, but what makes it the *right* one? The incontrovertible point is that the A vector a model happens to use has a degree of arbitrariness in it that makes using it for interpretation problematic. But what is the right criterion for picking an alternative “effective” version of A, A1? Being orthogonal to the null space is one well-defined choice, but is it clearly the right choice? (I believe this choice is also the one with minimal L2 norm. Is that important?) For example, Sec. 3.3 shows that raw attention, but not effective attention, gives evidence that attention shifts from [CLS] to [SEP] to periods and commas as L increases. This shows they are different, but what is the argument that the raw-attention conclusion is less ‘correct’ than the effective-attention conclusion here?
More generally, it would strengthen the paper even further to document more concrete cases where effective attention yields different interpretive conclusions than raw attention, especially to the extent that it can be argued that the former interpretations are somehow better than that latter.
Strengths.
The paper gives convincing evidence of the MAIN ARGUMENT’s propositions (P0) – (P3).
(P0) [Interpretation of the attention weight as a measure of the influence of one token on another] is frequent] Sec 4, 2nd paragraph, cites 16 papers to illustrate the point. While I have not examined all of them, even allowing for some possible disagreements of interpretation, I am confident there is ample relevant evidence among these papers and probably others as well.
(P1) [the attention weight itself includes an irrelevant quantity [C2]] The argument provided for this in Sec 3 seems correct to me, but as stated above as a ‘minor complaint concerning exposition’, it may not make as clear as possible the obviousness of the conclusion.
(P2) [Z(t,L) can be insufficient to determine which token t it corresponds to [C3]] This point is operationalized in Sec 4.2 by picking a function-approximation architecture G, and attempting to use it to perform the map Z(t,L) -> t, for a given L and for all input strings and tokens t. A linear and a non-linear MLP G are used. An experiment applying the pre-trained BERT-base Transformer to the Microsoft Research Paraphrase Corpus gives success rates that fall to 73% and 82% by final layer 12. That 18% of tokens cannot be recovered from their corresponding layer-12 encoding is the basis of the claim that equating attention from cell(t,L) to cell(t’,L) with attention from token t to token t’ is not justified in general.
That the non-linear approximator does 9% better than the linear one is the basis of the claim that ‘the non-linearities in BERT play an important role in maintaining token identifiability’ (Sec. 4.2). This claim should be spelled out explicitly. It reads to me to say that if the non-linearities in BERT were removed, token identifiability would be reduced, but the experiment does not show that. Rather, it shows that, in the presence of the non-linearities in BERT, a linear approximation to the token-identifying map G is significantly less accurate than the non-linear approximation: hardly a surprising conclusion. This bit of the argument needs to be clarified.
(P3) [Z(t,L) can contain strong contributions from tokens t’ =! t [C4]] As defined in Sec. 5, the contribution of token t’ to Z(t,L), for a given L, is proportional to the norm of the gradient of Z(t,L) with respect to the input from token t’; these are normalized to sum to 1. This is a transparent definition that resembles the previous gradient attribution proposal, but applied to internal rather than output representations. Using the same experimental set-up, it is shown that the average contribution of token t to Z(t,L) declines from 30% to 10% as L goes from 1 to 12. The proportion of values t such that the largest contribution to Z(t,L) is from a token t’ =! T rises from 0 for L = 1 through 3 to over 30% for L = 10 through 12. This is further evidence for the claim that equating attention between Z(t,L) and Z(t’,L) with attention between tokens t and t’ is problematic.


We present an architecture based on the conditional Variational Autoencoder to learn a representation of transformations in time-sequence data. The model is constructed in a way that allows to identify sub-spaces of features indicating changes between frames without learning features that are constant within a time-sequence. Therefore, the approach disentangles content from transformations. Different model-architectures are applied to affine image-transformations on MNIST as well as a car-racing video-game task. Results show that the model discovers relevant parameterizations, however, model architecture has a major impact on the feature-space. It turns out, that there is an advantage of only learning features describing change of state between images, over learning the states of the images at each frame. In this case, we do not only achieve higher accuracy but also more interpretable linear features. Our results also uncover the need for model architectures that combine global transformations with convolutional architectures.

Review of “Unsupervised-Learning of time-varying features”
This work looks at using a conditional VAE-based approach to model transformations of sequential data (transformations can be spatial, such as rotation of MNIST, or temporal, i.e. screenshots of a car racing game). Like how our own visual system encodes differences in time and space [1], they show that in generative modelling with VAEs, “there is an advantage of only learning features describing change of state between images, over learning the states of the images at each frame.”
Such an encoding allows the model to ignore features that are constant over time, and also makes it easier to represent data in a rotating curved manifold. They demonstrate this using data collected from CarRacing-v0 task (a task where agents have to learn to drive from pixel observation of a top-down track), and also on MNIST where the digits are rotated around the center of an image. They provide interesting analysis of the latent space learned and show that indeed this approach can handle both stationary and non-stationary features well (in CarRacing-v0). For MNIST, they compare the latent space learned from transformations (z_dot) and show that this approach can encode image geometric transformations quite well.
While this paper is interesting and highlights advantages of modeling transformations of sequential data, I don't think the contributions are currently sufficient for ICLR conference (right now it is a good workshop paper IMHO). For it to be at the conference level, I can make a few suggestions of things that will bring it there, hopefully the authors can take these points as feedback to help improve the work:
1) Would be great to see how this approach can compare to existing proposed algorithms (i.e. TD-VAE as cited)? Are there problems where this approach will perform really well that current methods are inadequate?
2) As the method is based on an RL-task, would the latent representation learned be useful for an RL agent that relies on the latent code across several representative RL tasks (in both sample efficiency, and/or terminal performance)?
I don't mean to discourage the authors (esp as an Anon Reviewer #2...), as I like the direction of the work, and also appreciate that a lot of effort has gone into this work. I hope to see the authors take the criticism to make their work better. Good luck!
[1] Concetta1988, Feature detection in human vision: A phase-dependent energy model


We present a new approach to defining a sequence loss function to train a summarizer by using a secondary encoder-decoder as a loss function, alleviating a shortcoming of word level training for sequence outputs. The technique is based on the intuition that if a summary is a good one, it should contain the most essential information from the original article, and therefore should itself be a good input sequence, in lieu of the original, from which a summary can be generated. We present experimental results where we apply this additional loss function to a general abstractive summarizer on a news summarization dataset. The result is an improvement in the ROUGE metric and an especially large improvement in human evaluations, suggesting enhanced performance that is competitive with specialized state-of-the-art models.

This paper proposes to use an additional component to the commonly used encoder-decoder approach for summarization, which is referred to as the recoder, which is an RNN-syle component that takes the output of the decoder. The intuition offered in the paper is that a good summary should produce itself via the recoder network, and in training it together with the original encoder-decoder it should improve its performance as it would be able to capture more than what the word-level loss does.
I have the following objections to this paper:
- I can't see why this extra component should improve the quality of the summary produced. If say our encoder-decoder architecture models the training data perfectly, then a recoder that does not do anything would be the right choice. Taking this further, a recoder could actually be fixing problems in the output of the decoder, and thus not providing a good training signal. It doesn't make sense to my that a summary should produce itself via a neural network, unless we are training an auto-encoder. The experiments validate this; the difference in ROUGE score are less than a point, which is the kind of fluctuation one expects due to random seed differences, etc.
- I find it odd that ROUGE score is dismissed as a loss to train against (referred to as a heuristic in the end of section 4.1), but then 1 ROUGE point difference is considered a "significant" improvement (making such claims without statistical significance testing is misleading). Sure it has flaws, if there is something better why not use it for evaluation? Furthermore, claiming that the recoder does a better job requires evidence. Why not train this extra function against human judgements? Assuming that what is wanted is to train a model that estimates the quality of the summary, it would make sense to look at the approaches used for the similar task of machine translation quality estimation: https://www.mitpressjournals.org/doi/pdf/10.1162/tacl_a_00056
- The approach is in my view a kind of actor-critic approach; the recoder could play that role, and in fact the Bahdanau et al. 2016 paper cited does this. However no comparison is offered, be it theoretical or experimental. Furthermore, the criticism that sequence-level training requires differentiable losses is incorrect; MIXER for example does train against scores such as ROUGE that are not differentiable. Furthermore, the BSO approach by Wiseman and Rush (2016) cited does give a continuous output to optimize for seq2seq that is asked for in the beginning of page 4.
- In the experiments, how was the length penalty determined to be the graduated curve mentioned in section 5? I would expect to have comparison against other approaches that try to train encoder-decoder to improve summarization, such as those mentioned in section 4.1.
- On the human evaluation experiments: the difference between the two models is quite small, especially given that the workers were not allowed to say that the models were equally good/bad. Furthermore, there is no inter-annotator agreement, as each comparison was done by a single crowd worker. Showing only the first 400 tokens of the original document would incorrectly disadvantage models selecting content from later in the articles. Finally, showing the reference summary creates another bias, since equally good summaries can disagree on what content to include. It might be helpful to look at some recent work on manual evaluation of summarization that tried to address these issues: https://arxiv.org/abs/1906.01361


Sparse neural networks have been shown to be more parameter and compute efficient compared to dense networks and in some cases they are even successfully used to decrease wall clock inference times. There is a large body of work on training dense networks to yield sparse networks for inference (Molchanov et al., 2017;Zhu & Gupta, 2018;Louizos et al., 2017;Guo et al., 2016). This limits the size of the largest trainable sparse model to that of the largest trainable dense model. In this paper we introduce a method to train sparse neural networks with a fixed parameter count and a fixed computational cost throughout training, without sacrificing accuracy relative to existing dense-to-sparse training methods. Our method updates the topology of the network during training by using parameter magnitudes and infrequent gradient calculations. We show that this approach requires fewer floating-point operations (FLOPs) to achieve a given level of accuracy compared to prior techniques. We demonstrate state-of-the-art sparse training results with ResNet-50, MobileNet v1 and MobileNet v2 on the ImageNet-2012 dataset, WideResNets on the CIFAR-10 dataset and RNNs on the WikiText-103 dataset. Finally, we provide some insights into why allowing the topology to change during the optimization can overcome local minima encountered when the topology remains static.

This paper proposes a pruning technique called RigL, which performs sparse initialization of the network weights, and allows the network to grow weights during training. The sparse initialization makes the pruning algorithm to be memory- and computation- efficient, unlike existing models that starts with dense network weights. The authors train RigL with three different sparsity distributions that considers the number of input and output nodes, and validate them on various deep convnets for training on ImageNet, on which the model outperforms several existing sparsification methods and even dense counterparts.
Pros
- The proposed model, RigL, is memory- and computation- efficient, and thus allows to train a large network in an efficient manner.

- RigL obtains impressive sparsification performance, even yielding sparse networks that outperform their dense counterparts.
Cons
- The idea of starting from a small, sparse network and expanding it is not novel. DEN [Yoon et al. 18] proposed the same bottom-up approach with sparsely initialized networks, while they allowed to increase the number of neurons at each layer and focused more on continual learning. The authors should compare the two methods both conceptually and experimentally.
- The method is more like a set of heuristics rather than a principled approach, which makes it less appealing. This is not really an issue if the paper includes extensive experimental validation and in-depth analysis, but this is not the case.

- The experimental validation is largely lacking, as the authors only perform experiments on ImageNet and do not compare against recent state-of-the-art Bayesian sparsification methods (SBP, VIB, L0-regularization). Without such extensive experimental validation, it is uncertain whether the result will generalize, given the highly empirical nature of the work.
In sum, although I believe that the paper proposes a very practical method that is easy to implement and is promising, due to lack of experimental validation against a similar approach, state-of-the-art sparsification methods, and results on more datasets, I temporarily provide the rating of weak reject. I may change my opinion if the authors provide those results during the rebuttal period.
[Yoon et al. 18] Lifelong learning with dynamically expandable networks, ICLR 2018