INFORMATION THEORETIC LOWER BOUNDS ON NEGATIVE LOG LIKELIHOOD
In this article we use rate-distortion theory, a branch of information theory devoted to the problem of lossy compression, to shed light on an important problem in latent variable modeling of data: is there room to improve the model? One way to address this question is to find an upper bound on the probability (equivalently a lower bound on the negative log likelihood) that the model can assign to some data as one varies the prior and/or the likelihood function in a latent variable model. The core of our contribution is to formally show that the problem of optimizing priors in latent variable models is exactly an instance of the variational optimization problem that information theorists solve when computing ratedistortion functions, and then to use this to derive a lower bound on negative log likelihood. Moreover, we will show that if changing the prior can improve the log likelihood, then there is a way to change the likelihood function instead and attain the same log likelihood, and thus rate-distortion theory is of relevance to both optimizing priors as well as optimizing likelihood functions. We will experimentally argue for the usefulness of quantities derived from rate-distortion theory in latent variable modeling by applying them to a problem in image modeling.

(Please find my response to the rebuttal and updated version in a comment below)
The paper analyses latent-variable modeling from a rate-distortion point-of-view in a novel and interesting fashion, highlighting important fundamental connections. In particular, the paper presents a novel theorem (inspired by how the rate-distortion function is computed) that gives a lower bound on the negative log likelihood. This lower bound allows to quantify by how much a latent-variable model could be improved by either modifying the prior or the likelihood function. The latter is important, since the paper shows a duality between improving one while keeping the other fixed and vice versa. Finally, the paper derives a practical implementation/approximation (founded on solid theoretical analysis) of quantifying the improvement potential of a latent-variable model. These, so called “glossy statistics” are quantitatively analyzed in a set of experiments with different variational autoencoder architectures (various likelihood models and priors) on a number of datasets.
The main contribution of this paper is to provide novel proofs and theoretical analysis that connect latent-variable modeling with rate-distortion on a very fundamental level. While similar attempts have been reported in the recent literature (perhaps a bit more focused on the empirical aspects), the analysis and results in the paper follow a very fundamental treatment of rate-distortion theory and in particular of computation of the rate-distortion function. The central idea underlying rate-distortion, i.e. lossy compression by discarding irrelevant information, seems very suitable as a guiding principle for representation learning. In particular, learning representations that generalize well is essentially another instance of a lossy compression problem. The paper thus addresses an important and timely topic which should be of broad interest to the representation learning community. The paper is well written and mathematically rigorous. I have checked most parts of the proofs, though there still is a chance that I missed something. I am not entirely convinced by the practical impact of the experimental section of the paper (though the experiments are beyond toy-level and I do not doubt the results), but I also believe that this is not the main contribution of the paper, which is rather laying the mathematical groundwork for future work. I vote and argue for accepting the paper for presentation at the conference. My criticism below is aimed at giving some pointers for potentially improving the paper.
1) As the paper acknowledges, there is a risk of overfitting when improving likelihood functions under fixed priors (and vice versa). While the glossy statistics certainly allow making approximate statements of whether the model can be improved further or not, there is no “threshold value” or other guideline that would indicate a modeling expert that they are entering an over-fitting regime if the model-class is further enriched. Therefore, I am not sure about the practical impact of the experiments: the glossy statistics seem to be indicative of the margin for improvements in the negative log-likelihood - but whether all of these improvements are really desirable is unclear. To test this, one might resort to tasks other than generative modeling, such that models that overfit can easily be “spotted” (by degrading test-set performance).
2) Rate-Distortion can be “made more robust” against overfitting by different choices of \alpha (essentially limiting channel capacity). Maybe I am missing something, but shouldn’t the \alpha carry over into the computation of the ratios for c(z)? Was it just assumed to be 1? The same question for Theorem 2 and the equation just above Theorem 2 - does the alpha drop (is it absorbed into the likelihood) or was it set to one? It might be interesting to see how the glossy statistics behave if \alpha is considered a hyper-parameter of the model, e.g. under “low capacity” do the glossy statistics “flatten out” very early?
3) I would have been excited to see how the glossy statistics evolve during training of a model - it would be interesting to show that the statistics initially predict a large margin of improvement that reduces and slowly flattens out as training converges.
4) In the paragraph after Eq. 14: the argument hinges on the possibility of having an invertible (and continuously differentiable) g(z). To me it is not straightforward that a neural network would necessarily implement such a function (particularly the invertibility might be problematic). Is this just a technical condition required for the formal statement, or do you think that this issue could become problematic in practice as well such that the duality between improving prior and likelihood does not hold any longer?
Minor:
5) I think the Alemi et al. reference (first reference) has been published under a different name (Fixing a Broken ELBO) at this years’ ICML.
6) Consider calling the quantity l(x|z) below Eq. 1 “the likelihood of the latent variable given the data” (since the data is given, even though the data is not in the conditional, which is why it is a likelihood function).
7) Rather than using “the KL divergence between”, use “the KL divergence from … to” which nicely reflects its asymmetry.
8) Page 4, last Equation: square brackets for E_X missing


Text classification must sometimes be applied in situations with no training data in a target language. However, training data may be available in a related language. We introduce a cross-lingual document classification framework (CACO) between related language pairs. To best use limited training data, our transfer learning scheme exploits cross-lingual subword similarity by jointly training a character-based embedder and a word-based classifier. The embedder derives vector representations for input words from their written forms, and the classifier makes predictions based on the word vectors. We use a joint character representation for both the source language and the target language, which allows the embedder to generalize knowledge about source language words to target language words with similar forms. We propose a multi-task objective that can further improve the model if additional cross-lingual or monolingual resources are available. CACO models trained under low-resource settings rival cross-lingual word embedding models trained under high-resource settings on related language pairs.

Overview:
This paper proposes an approach to document classification in a low-resource language using transfer learning from a related higher-resource language. For the case where limited resources are available in the target low-resource language (e.g. a dictionary, pretrained embeddings, parallel text), multi-task learning is incorporated into the model. The approach is evaluated in terms of document classification performance using several combinations of source and target language.
Main strengths:
1. The paper is well written. The model description in Section 2 is very clear and precise.
2. The proposed approach is simple but still shows good performance compared to models trained on corpora and dictionaries in the target language.
3. A large number of empirical experiments are performed to analyse different aspects and the benefits of different target-language resources for multi-task learning.
Main weaknesses:
1. The application of this model to document classification seems to be new (I am not a direct expert in document classification), but the model itself and the components are not (sequence models, transfer learning and multitask learning are well-established). So this raises a concern about novelty (although the experimental results are new).
2. With regards to the experiments, it is stated repeatedly that the DAN model which are compared to uses "far more resources." The best ALL-CACO model also relies on several annotated but "smaller" resources (dictionaries, parallel text, embeddings). Would it be possible to have a baseline where a target-language model is trained on only a small amount of annotated in-domain document classification data in the target language? I am proposing this baseline in order to answer two questions. (i) Given a small amount of in-domain data for the task at hand, how much benefit do we get from additionally using data from a related language? (ii) How much benefit do we get from using target-language resources that do not address the task directly (dictionaries, embeddings) compared with using a "similar" amount of data from the specific task?
Overall feedback:
This is a well-written paper, but I think since the core of the paper lies in its empirical evaluation, the above experiments (or something similar) would greatly strengthen the work.
Edit: I am changing my rating from 5 to 6 based on the authors' response.


Convolutional Neural Networks (CNN) are very popular in many fields including computer vision, speech recognition, natural language processing, to name a few. Though deep learning leads to ground breaking performance in these domains, the networks used are very demanding computationally and are far from real-time even on a GPU, which is not power efficient and therefore does not suit low power systems such as mobile devices. To overcome this challenge, some solutions have been proposed for quantizing the weights and activations of these networks, which accelerate the runtime significantly. Yet, this acceleration comes at the cost of a larger error. The NICE method proposed in this work trains quantized neural networks by noise injection and a learned clamping, which improve the accuracy. This leads to state-of-the-art results on various regression and classification tasks, e.g., ImageNet classification with architectures such as ResNet-18/34/50 with low as 3-bit weights and 3 -bit activations. We implement the proposed solution on an FPGA to demonstrate its applicability for low power real-time applications.

The authors present a method for fine-tuning neural networks so inference can be performed in a quantized low bit data format down to 3 bits. The authors achieve this through a combination of three techniques:
1. Noise injection to fine-tune the weights before quantization. The effect of noise injection can model that of quantization, but rather than being stuck in a quantization bin, fine grained weight updates are still possible
2. A schedule that quantizes layer by layer, rather than all layers at the same time
3. Clipping weights and activations within a learned range to obtain finer grained bins within that range.
The main contribution is a novel combination of mostly existing techniques. Clipping (or clamping as the authors call it) has been proposed by Zhang et al. 2018, but it's an interesting contribution to have the clipping learned directly via backpropagation with a straight-through estimator. Treating the quantization as noise has been proposed in a different form in McKinstry et al. 2018. Gradual quantization appears novel, but is also the least interesting of the techniques. Therefore, novelty on ideas/methods is somewhat limited, and the contribution is mostly the in the impressive experimental results, which appear to be outperforming previous methods. The main weaknesses are poor writing, and that some details of the implementation required to reproduce the results are missing. For example, the training schedule is not given, e.g. how many epochs to train the clean model, how many with noise, how many quantized. Details on the gradual quantization are also missing. Block based quantization is completely heuristic and not well motivated. If this is the main novel ingredient, more details on the mechanics would be needed. Is both the noise injection and the quantization done in blocks? If the motivation is in "the opportunity to adapt”, then what does the adaptation look like?
As above, my other main issue is with the writing, there are many examples where I would suggest improvements:
This work could be improved greatly by copy editing for English grammar. There are many typos (including ones that can be caught by autocorrect, missing punctuation, or using similar but unrelated words, e.g. "token" instead of "taken"). The manuscript appears hastily put together and not ready for publication.
The acronym NICE already has a meaning in the DL literature: Dinh, L., Krueger, D., & Bengio, Y. (2014). NICE: Non-linear Independent Components Estimation. It confusing to reuse it.
The term clamping is only explained on page 4 but used since the abstract. It’s used in a nonstandard way to mean “constrained to lie within a range” which should be explained earlier. I think “clipped” would be a better term, following the related Choi et al. 2018. Clamping usually means "constrained to a fixed value" (not a range), so it is not a good term to use in this context.
Are the results shown in table 2 and table 3 from a single trial or averaged across reruns? If single trial, it's misleading to have 2 figures after the decimal. Even non-quantized ResNet tends to have 0.5% or so run to run variability, which is much larger than the differences between some of the methods shown here. In fact, a lot of the results could just be due to picking a lucky random seed.
Comparisons are shown against methods JOINT (Jung et al), LQ-Nets (Zhang et al), FAQ (McKinstry et al). It would be helpful to present them with the same names in the "related work" section, and explain why they were picked out for the comparison. For someone not familiar with the literature it's hard to see why these 3 would be the obvious picks.
Readability would increase if table 2 and 3 were moved to section 4 where they are referenced, rather than after the discussion. Fig 2 font size too small and hard to read.


Experience replay (ER) is crucial for attaining high data-efficiency in off-policy deep reinforcement learning (RL). ER entails the recall of experiences obtained in past iterations to compute gradient estimates for the current policy. However, the accuracy of such updates may deteriorate when the policy diverges from past behaviors, possibly undermining the effectiveness of ER. Previous off-policy RL algorithms mitigated this issue by tuning their hyper-parameters in order to abate policy changes. We propose ReF-ER, a method for active management of experiences in the Replay Memory (RM). ReF-ER forgets experiences that would be too unlikely with the current policy and constrains policy changes within a trust region of the behaviors in the RM. We couple ReF-ER with Q-learning, deterministic policy gradient and off-policy gradient methods to show that ReF-ER reliably improves the performance of continuous-action off-policy RL. We complement ReF-ER with a novel off-policy actor-critic algorithm (RACER) for continuous-action control. RACER employs a computationally efficient closed-form approximation of the action values and is shown to be highly competitive with state-of-the-art algorithms on benchmark problems, while being robust to large hyper-parameter variations.

The authors introduce two new algorithms: remember and forget experience replay (ReF-ER), and an actor-critic architecture for continuous-action problems which is significantly more computationally efficient than previous approaches (RACER). ReF-ER manages the experience in the replay memory more directly and removes trajectories (episodes) that follow policies less related to the current parameterized policy (based on the importance weights). RACER's main contribution is provides a closed form approximation of the action values, enabling significant gains computationally. They provide several empirical studies in benchmark domains showing the competitiveness of their approach, and the provided more stability to various continuous control algorithms (NAF, PG, u-DDPG).
Overall, I think it is a nicely written paper with a lot of empirical evidence of the usefulness of ReF-ER. I am quite interested in this algorithm specifically, as the active management of experience in the replay memory is an important step towards the ER acting as a proxy to short term memory. To my knowledge this algorithm is novel, and performs admirably. I'm less clear of the main benefits of RACER over previous approaches, except for better computational complexity. This primarily comes from a lack of empirical comparison, and not much explanation as to why key competitors were excluded. The inclusion of RACER seems to muddy the message of the paper, and a much stronger and deeper look at ReF-ER would have made for a stronger submission.
I have several questions for clarity and more comments below, but overall I think the paper is quite useful for the community and contains interesting insight into active management of transitions in an experience replay buffer.
Pros:
------
Lots of empirical studies. And a lot of details to impart intuition of the new experience replay.
Interesting take on experience replay.
Convincing results in many simulation benchmark domains (even though the competitors are sparse).
Cons:
------
There is some ambiguity and maybe some confusion about the difference between control and off-policy learning. While I agree you are learning off-policy for control (due to the experience replay buffer containing old data), the terms off-policy and on-policy seem overused here. Statements such as "ER has become one of the mainstay techniques to improve the sample-efficiency of off-policy RL" aren't entirely correct as the experience replay buffer is primarily used in deep reinforcement learning to improve sample-efficiency, not off-policy reinforcement learning as a whole.
The RACER algorithm seems to muddy up the message of the paper quite a bit. I would have much preferred an in-depth look at ReF-ER here, rather than the introduction of two algorithms. And I think your paper would have been stronger for it. That being said, the RACER algorithm seems incomplete. While it is an improvement over prior approachers (ACER) computationally, the need to use ReF-ER is concerning. I'm also a bit confused why ACER isn't used as a competitor against RACER? Even if you aren't outperforming the other approach on all benchmarks, the improved computational complexity is still a worthwhile improvement.
No confidence bounds in the results, although these are somewhat shown in the appendix (without the competitors shown!!). I'm curious at the significance of the different parameter settings.
Questions:
----------
I'm curious as to how this is related to something like rejection sampling? Or other importance sampling approaches more directly? How does your method compare with using retrace or some other off-policy algorithms? I'm unclear on the reasons why these types of comparisons aren't made empirically, could you clarify more directly?
Does your algorithm help with variance issues of other off-policy algorithms? Such as just using importance weights instead of retrace? How would it effect tree backup or just the usual importance sampling? It seems likely that this would help here, as you are limiting the amount of data with high importance weights, although this might also add bias.
Have you removed the target network in your experiments? This detail is not obvious in the paper currently and when you introduce ReF-ER you seem to be leading to this, but never say explicitly.
You claim that ReF-ER "reduces the sensitivity on the network architecture and training hyper-parameters." I'm unclear how you show this in the results with the current paper. You do some hyperparameter studies in the appendix, but don't compare against other algorithms here. Could you share a bit further how you are measuring the sensitivities of your algorithm against the competitors?
Do you need to anneal the cmax? What are the effects if this is set to some constant?
Could you expand on the results of HumanoidStandup-v2? Why do you believe your approach does significantly worse than the baselines here?
For DDPG, what happens if you change the bounds instead of removing them entirely? Also how does your method compare on a domain without unbounded actions?
It is unclear why RACER does not work with ER/PER. Do you have any intuition here? Could this be fixed through means other than ReF-ER?
Other minor comments (not taken into consideration for the review):
-------
Pseudo code: It is a bit unclear what algorithm 1 is supposed to be, I'm assuming ReF-ER?
Begin revision comments:
-----
Given the revisions to this paper, I am more confident that it will be of interest to the community. The major contributions here I see is the removal of target networks given their approach. Given this I still have concerns on clarity and still am unhappy with the lack of confidence intervals in the main experimental section. I've increased my score to 7 to reflect my increase in confidence.


Modern convolutional networks are not shift-invariant, despite their convolutional nature: small shifts in the input can cause drastic changes in the internal feature maps and output. In this paper, we isolate the cause -the downsampling operation in convolutional and pooling layers -and apply the appropriate signal processing fix -low-pass filtering before downsampling. This simple architectural modification boosts the shift-equivariance of the internal representations and consequently, shift-invariance of the output. Importantly, this is achieved while maintaining downstream classification performance. In addition, incorporating the inductive bias of shift-invariance largely removes the need for shift-based data augmentation. Lastly, we observe that the modification induces spatially-smoother learned convolutional kernels. Our results suggest that this classical signal processing technique has a place in modern deep networks.

This work shows that adding a simple blurring into max pooling layers can address issues of image classification instability under small image shifts. In general this work presents a simple and easy to implement solution to a common problem of CNNs and even though it lacks more thorough theoretical analysis of this problem from the signal processing perspective (such as minimal size of the blurring kernel for fulfilling the Nyquist-Shannon sampling theorem), it seems to provide ample empirical evidence.
Pros:
+ The introduction and motivation is really well written and Figure 3 provides a clear visualisation main max pooling operator issues.
+ The proposed method is really simple and shows promising results on the CIFAR dataset. With random shifts, authors had to tackle cropping with circular shifts. As it can cause artifacts in the data, authors also provide baseline performances on the original data (used for both training and testing).
+ Authors provide a thorough evaluation, ranging from comparing hidden representations to defining consistency metrics of the classified classes.
This work is lacking in the experimental section due to some missing details and few inconsistencies. I believe the most of my concerns can be relatively easily fixed/clarified in an update of this submission.
Major issues, which if fixed would improve the rating:
- It is not correct to average test accuracy and test consistency as both measures are different quantities, especially when using them for ranking. The difference between accuracy of different methods are considerably smaller than differences in the classification consistency.
- It is not clear how many shifts are used for computing the "Random Test Accuracy" and the "Classification Accuracy". Also whether the random shifts are kept constant between evaluated networks and evaluation metrics.
- Authors do not address the question what is the correct order of operations for the blurring. E.g. would the method empirically work if blurring was applied before max pooling? Do the operations commute?
- The selection of the filters is rather arbitrary, especially regarding the 1D FIR filters. The separability of these filters should be discussed.
- I believe authors should address how this work differs to [1], as it also tests different windowing functions for pooling operators, even though in different tasks.
Minor issues, which would be nice to fix however which do not influence my rating:
* Section 3.1 - And L-Layer deep *CNN*, H_l x W x C_l -> H_l x W_l x C_l
* Section 3.1. Last paragraph - I would not agree with the statement that in CNNs the shift invariance must necessarily emerge upon shift equivariance. If anything, this may hold only for the last layer of a network without fully connected layers and with average pooling of the classifier output (ResNet/GoogleNet like networks).
* Explicitly provide the network architecture as [Simonyan14] does not test on CIFAR and cannot use Batch normalisation.
* It would be useful to add citation for the selected FIR filters.
* The flow of section 4.2. can be improved to help readability. The three metrics should be first motivated before their introduction. Metric 2. paragraph - the metric is defined below, not above.
* It would be interesting to see what would be the performance if the blurring filters were trained as well (given some sensible initialisation).
* One future direction would be to verify that this approach generalises to larger networks as well. It might be worth to discuss this in the conclusions.
[1] Scherer, Dominik, Andreas Müller, and Sven Behnke. "Evaluation of pooling operations in convolutional architectures for object recognition." Artificial Neural Networks–ICANN 2010. Springer, Berlin, Heidelberg, 2010. 92-101.


Graph convolutional networks have achieved tremendous success in the tasks of graph node classification. These models could learn a better node representation through encoding the graph structure and node features. However, the correlation between the node labels are not considered. In this paper, we propose a novel architecture for graph node classification, named conditional graph neural fields (CGNF). By integrating the conditional random fields (CRF) in the graph convolutional networks, we explicitly model a joint probability of the entire set of node labels, thus taking advantage of neighborhood label information in the node label prediction task. Our model could have both the representation capacity of graph neural networks and the prediction power of CRFs. Experiments on several graph datasets demonstrate effectiveness of CGNF.

This paper proposed a combination of graph neural networks and conditional random field to model the correlation between node labels in the output.  In typical graph neural nets the predictions for nodes are conditionally independent given the node representations.  This paper proposes to use a CRF to compensate for that.  In terms of the approach, the authors used GCNs to produce unary potentials for the CRF, and have the pairwise potentials on each edge to model the correlation of labels of neighboring nodes.  Learning is done by optimizing pseudo-likelihood and the energy loss, while inference is performed through a couple heuristic processes.
Combining neural nets with CRFs is not a new idea, in particular this has been tried before on image and sequence CRFs.  It is therefore not surprising to see an attempt to also try it for graph predictions.  The main argument for using a CRF is its ability to model the correlations of output labels which was typically treated as independent.  However this is not the case for deep neural networks, as it already fuses information from all over the input, and therefore for most prediction problems it is fine to be conditionally independent for the output, as the dependence is already modeled in the representations.  This is true for graph neural networks as well, if we have a deep graph neural net, then the GNN itself will take care of most of the dependencies between nodes and produce node representations that are suitable for conditionally independent output predictions.  Therefore I’m not convinced that CRFs are really necessary for solving the prediction tasks tried in this paper.
The learning and inference algorithms proposed in this paper are also not very convincing.  CRFs has been studied for a long time, and there are many mature algorithms for learning them.  We could do proper maximum conditional likelihood learning, and use belief propagation to estimate the marginals to compute the gradients.  Zheng et al. (2015) did this for convnets, we could also do this for graph CRFs as belief propagation can be easily converted into message passing steps in the graph neural network.  Pseudo-likelihood training makes some sense, but energy loss minimization doesn’t really make sense and has serious known issues.
On the other hand, the proposed inference algorithms does not have good justifications.  Why not use something standard, like belief propagation for inference again?  Our community has studied graphical models a lot in the last decade and we have better algorithms than the ones proposed in this paper.
Lastly, the experiments are done on some standard but small benchmarks, and my personal experience with these datasets are that it is very easy to overfit, and most of the effort will be put in to prevent overfitting.  Therefore more powerful models typically cannot be separated from overly simple models.  I personally don’t care a lot about the results reported on these datasets.  Besides, there are a lot of questions about the proposed model, but all we get from the experiment section are a few numbers on the benchmarks.  I expect studies about this model from more angles.  One more minor thing about the experiment results: the numbers for GraphSAGE are definitely wrong.
Overall I think this paper tackles a potentially interesting problem, but it isn’t yet enough to be published at ICLR due to its problems mentioned above.


GAMEPAD: A LEARNING ENVIRONMENT FOR THEO- REM PROVING
In this paper, we introduce a system called GamePad that can be used to explore the application of machine learning methods to theorem proving in the Coq proof assistant. Interactive theorem provers such as Coq enable users to construct machine-checkable proofs in a step-by-step manner. Hence, they provide an opportunity to explore theorem proving with human supervision. We use GamePad to synthesize proofs for a simple algebraic rewrite problem and train baseline models for a formalization of the Feit-Thompson theorem. We address position evaluation (i.e., predict the number of proof steps left) and tactic prediction (i.e., predict the next proof step) tasks, which arise naturally in tactic-based theorem proving.

Summary: This paper mixes automated theorem proving with machine learning models. The final goal, of course, is to be able to train a model that works in conjunction with an automated theorem proving system to efficiently prove theorems, and, ideally, in a way that resembles the way humans prove theorems. This is a distant goal, and the authors instead focus on several tractable tasks that are required for future progress in this direction. They start by integrating the Coq theorem proving environment with ML frameworks, allowing for the creation of models that perform various tasks related to theorem proving. In particular, they focus on two tasks. One is to estimate how many steps are left to complete the proof given a current proof state. The other is to determine what is a good choice of next step. Finally, they also consider issues surrounding representations of the various data structures involved in proofs (i.e., the proof tree, variables, etc.). They test various models on a synthetic nearly trivial logical expression proof, along with a more complicated (and meaningful real world) group theory result.
Strengths: This is a very important area. Automated theorem proving has a potentially very significant impact, and being able to take advantage of some of the recent successes in ML would be excellent. The main environment proposed here, integrating PyTorch with Coq could potentially be a very useful platform for future research in this area. The paper exposes many interesting questions, and I generally think we need more exploratory papers that open up an area (as opposed to seeking to finalize existing areas)
Weaknesses: The paper is pretty tough to understand without a lot of background in all of the existing theorem proving work (which might be fine for a conference in this area, but for this venue it would be nice to be more self-contained). The organization could also use some work, since it's often tough to figure out what the authors actually did. The experimental results seem very preliminary---although it's hard to say, as there is no easy way to compare the results to anything else out there. In general a lot of details seem missing.
Verdict: The authors admit this is a preliminary work, and I agree with that. The paper certainly introduces many more questions than it answers. However, I think that in this case it's a good thing, and this type of paper has the potential to inspire a lot of new and exciting research, so I voted for acceptance.
Comments and questions:
- As mentioned, a lot of the terminology is introduced very quickly and could stand to be more self-contained, i.e., "tactics" could be defined as being simple transformations that are applied to a current proof state to obtain another proof state, and each language has a library of tactics available.
- Probably the major contribution of the work is the integration of the CoQ and Pytorch, so a bit more content describing how the Python data structures that wrap around Coq structures would be interesting here.
- I didn't really understand one of the major contributions: the embedding function for the M_i conditioned on the environment. How does the sampled Gaussian vector work here? In general this section is pretty confusing, it would be great to include a schematic to show how the different levels of embeddings for different structures work here.
- How does the real-world dataset work? Does the dataset contain one automated proof of the entire theorem, or several different proofs (ultimately produced by different user choices)? Are you measuring accuracy on the proofs of individual lemmas?


PATE-GAN: GENERATING SYNTHETIC DATA WITH DIFFERENTIAL PRIVACY GUARANTEES
Machine learning has the potential to assist many communities in using the large datasets that are becoming more and more available. Unfortunately, much of that potential is not being realized because it would require sharing data in a way that compromises privacy. In this paper, we investigate a method for ensuring (differential) privacy of the generator of the Generative Adversarial Nets (GAN) framework. The resulting model can be used for generating synthetic data on which algorithms can be trained and validated, and on which competitions can be conducted, without compromising the privacy of the original dataset. Our method modifies the Private Aggregation of Teacher Ensembles (PATE) framework and applies it to GANs. Our modified framework (which we call PATE-GAN) allows us to tightly bound the influence of any individual sample on the model, resulting in tight differential privacy guarantees and thus an improved performance over models with the same guarantees. We also look at measuring the quality of synthetic data from a new angle; we assert that for the synthetic data to be useful for machine learning researchers, the relative performance of two algorithms (trained and tested) on the synthetic dataset should be the same as their relative performance (when trained and tested) on the original dataset. Our experiments, on various datasets, demonstrate that PATE-GAN consistently outperforms the stateof-the-art method with respect to this and other notions of synthetic data quality.

[Post revision update] The authors' comments addressed my concerns, especially on the experiment side. I changed the score.
This paper applies the PATE framework to GAN, and evaluates the quality of the generated data with some predictive tasks. The experimental results on some real datasets show that the proposed algorithm outperforms DPGAN, and the generated synthetic data is quite useful in comparison with real data.
The presentation is clear and easy to follow. However, I think the paper needs to be improved in its novelty, and the techniques and experiments need to be more thorough.
More details:
- It might be necessary to consider using Gaussian noise[24] in replace of the Laplace noise, which, according to [24], would improve privacy and accuracy.
- This paper:
“Privacy-preserving generative deep neural networks support clinical data sharing” by Brett K. Beaulieu-Jones, Zhiwei Steven Wu, Chris Williams, Casey S. Greene
seems quite relevant. If so, you may want to add some discussion in the related work section or compare with their result.
- The last paragraph of the related works section mentioned some related work with shortcomings as working only on low-dimensional data and features of specific types, yet the experiments are also mostly done on low-dimensional datasets. I think it would be better to do a thorough evaluation on data of different kinds, such as image data.
- If the two evaluation metrics for private GAN is considered an important contribution of the paper, it might be better to make it a separate section and elaborate more on the motivation and method.
- It might be better to move some details (for example, instead of presenting the results of the 12 predictive models, presenting only the average, as it’s not very important how each of them performs) of the credit card fraud detection dataset to the appendix and bring the results of the other datasets to the main body.


STOCHASTIC PREDICTION OF MULTI-AGENT INTERACTIONS FROM PARTIAL OBSERVATIONS
We present a method that learns to integrate temporal information, from a learned dynamics model, with ambiguous visual information, from a learned vision model, in the context of interacting agents. Our method is based on a graph-structured variational recurrent neural network (Graph-VRNN), which is trained end-to-end to infer the current state of the (partially observed) world, as well as to forecast future states. We show that our method outperforms various baselines on two sports datasets, one based on real basketball trajectories, and one generated by a soccer game engine.

1) Summary
This paper presents a graph neural network based architecture that is trained to locate and model the interactions of agents in an environment directly from pixels. They propose an architecture that is a composition of recurrent neural networks where each models a single object independently and communicate with other for the overall environment modeling. The model is trained with a variational recurrent neural network objective that allows for stochasticity in the predictions while at the same time allows to model the current and future steps simultaneously. In experiments, they show the advantage of using the proposed model for tasks of tracking as well as forecasting of agents locations.
2) Pros:
+ Novel recurrent neural network architecture to model structured dynamics of agents in an environment.
+ Outperforms baseline methods.
+ New dataset for partially observable prediction research.
3) Cons:
Forecasting task:
- The authors argue that a discretization needs to be performed because of the many possible futures given the past, and also provide an error measure based on likelihood. However, if trajectories are actually generated from these distributions, I suspect the many possible futures generated will be very shaky. Can the authors provide trajectories sampled from this? If sampling trajectories does not make sense somehow, can the authors comment on how we can sample multiple trajectories?
Lack of baselines:
- The authors mention social LSTM and social GAN in the related work, however, no comparison is provided. From a quick glance, the authors of these papers work on trajectories. However, the “social” principle in those papers is general since it’s done from the computed feature vector. Could it have not been used on top of one of the baselines? If not, could the authors provide a reason why this is not the case?
Additional comments:
As the authors mention, it would be nice to extend this paper to an unsupervised or semi-supervised task. Here are a couple of papers that may interest you:
https://arxiv.org/abs/1804.04412
https://arxiv.org/abs/1705.02193
https://arxiv.org/abs/1806.07823
4) Conclusion
Overall, the paper is well written, easy to understand, and seems to be simple enough to quickly reproduce. Additionally, the proposed dataset may be of use for the community. If the authors are able to successfully address the issues mentioned, I am willing to improve my score.


Neural language models (NLMs) exist in an accuracy-efficiency tradeoff space where better perplexity typically comes at the cost of greater computation complexity. In a software keyboard application on mobile devices, this translates into higher power consumption and shorter battery life. This paper represents the first attempt, to our knowledge, in exploring accuracy-efficiency tradeoffs for NLMs. Building on quasi-recurrent neural networks (QRNNs), we apply pruning techniques to provide a "knob" to select different operating points. In addition, we propose a simple technique to recover some perplexity using a negligible amount of memory. Our empirical evaluations consider both perplexity as well as energy consumption on a Raspberry Pi, where we demonstrate which methods provide the best perplexity-power consumption operating point. At one operating point, one of the techniques is able to provide energy savings of 40% over the state of the art with only a 17% relative increase in perplexity.

In this paper, the authors investigate the accuracy-efficiency tradeoff for neural language models. In particular, they explore how different compression strategies impact the accuracy (and flops), and more interestingly, also how it impacts the power use for a RaspberryPi. The authors consider the QRNNs and SRUs for this purpose and use standard datasets for their analysis. I am torn about this paper. On one hand, I feel that the analysis is interesting, thoughtful and detailed; the power usage statistics bring a different perspective to the compression community. The section on inference time pruning was especially interesting to read. On the other hand however, there is limited novelty in the setup. The authors use standard, well known, compression algorithms on common neural language modeling architectures and datasets and use out-of-the-box tools for their ultimate analysis. Further, the paper needs additional work before it can be accepted in my opinion. I detail my arguments below:
- The authors begin by discussing SwiftKey and similar apps but I'm not sure if its clear that they use neural language modeling as the backend. Do the authors have a source to validate this claim?
- Knowledge distillation is another algorithm that has been found to be quite competitive in compressing models into smaller versions of themselves. Have the authors experimented with that?
- The R@3 is an good metric but I suggest that the authors look at mean reciprocal rank (MRR) instead. This removes the arbitrary-ness of "3" while ensuring that the metric of interest is the accuracy and not probability of being correct (perplexity).
- Can you comment on the sensitivity of the results to the RPi frameworks? For instance, the RPi deployment tools, architecture, and variance in the predictions?
- Along the same line, I'm curious how generalizable the RPi results are for other computing architectures. For those of us who are not experts on hardware, it would be nice to read about whether similar tradeoffs will exist in other architectures such as mobile phones, GPUs or CPUs.
- Could the authors add some meta-analysis about the results? If the perplexity goes up as a consequence of compression, what kinds of tokens it that localized to? Is it primarily rare words that the model is less confident about, or are the probabilities for most words getting skewed?
- Finally, I feel that such an exploration will catch on only if the tools are open-sourced and made easy to replicate/use. If there were a blog or article summarizing the steps needed to replicate the power measurement (including sources from where to buy the necessary hardware), more people would be inclined on adding such an analysis to future neural language modeling work.
I am willing to revisit my rating, as necessary, once I read through the rebuttal.
UPDATE:
After reading the rebuttal, I am increasing my score to 6. The authors alleviated some of my concerns but my major concerns about their novelty and the impact of their results remains.


We present a framework for interpretable continual learning (ICL). We show that explanations of previously performed tasks can be used to improve performance on future tasks. ICL generates a good explanation of a finished task, then uses this to focus attention on what is important when facing a new task. The ICL idea is general and may be applied to many continual learning approaches. Here we focus on the variational continual learning framework to take advantage of its flexibility and efficacy in overcoming catastrophic forgetting. We use saliency maps to provide explanations of performed tasks and propose a new metric to assess their quality. Experiments show that ICL achieves state-of-the-art results in terms of overall continual learning performance as measured by average classification accuracy, and also in terms of its explanations, which are assessed qualitatively and quantitatively using the proposed metric.

Summary:
In this paper, the authors propose a framework for continual learning based on explanations for performed classifications of previously learned tasks. In this framework, an average saliency map is computed for all images in the test set of a previous task to identify image regions, which are important for that task. When learning the next task, this average saliency map is used in an attention mechanism to help learning the new task and to prevent catastrophic forgetting of previously learned tasks. Furthermore, the authors propose a new metric for the goodness of a saliency map by taking into account the number of pixels in the map, the average distance between pixels in the map, as well as the prediction probability given only the salient pixels.
The authors report that their approach achieves the best average classification accuracy for 3 out of 4 benchmark datasets compared to other state-of-the-art approaches.
Relevance:
This work is relevant to researchers in the field of continual/life-long learning, since it proposes a framework, which should be possible to integrate into different approaches in this field.
Significance:
The proposed work is significant, since it explores a new direction of using learner generated, interpretable explanations of the currently learned task as help for learning new tasks. Furthermore, it proposes a new metric for the goodness of saliency maps.
Soundness:
In general, the proposed approach of using the average saliency map as attention mask for learning appears to be reasonable. However, the following implicit assumptions/limitations of the approach should be made more clear:
	- important features for the new task should be in similar locations as important features of the old task (for example, one would expect that the proposed approach would negatively affect learning the new task if the important features of the old task were all located in the bottom of the image, while all important features for the new task are in the top)
	- the locations for important features should be comparatively stable (for example, one would expect the average saliency map to become fairly meaningless if important features, such as the face of a dog, can appear anywhere in the image. Therefore, an interesting baseline for the evaluation of the ICL approach would be a predefined, fixed attention map consisting of concentric circles with the image center as their center, to show that the proposed approach does more than just deemphasizing the corners of the image)
Furthermore, the authors appear to imply that increased FSM values for an old task after training on a new task indicate catastrophic forgetting. While this is a reasonable assumption, it does not necessarily seem to be the case that a larger, more disconnected saliency map indicates worse classification performance. Comparatively small changes in FSM may not affect the classification performance at all, while larger changes may not necessarily lead to worse classifications either. For example, by increasing the amount or size of image regions to be considered, the classifier may accidentally become more robust on an old task. Therefore, it may be a good idea for the authors to analyze the correlation between FSM changes and accuracy changes.
Evaluation:
The evaluation of the proposed approach on the four used datasets appears to be reasonable and well done. However, given that the achieved performance gains over the state-of-the-art are fairly small, it would be good to assess if the obtained improvements are statistically significant.
Furthermore, it may be informative to show the saliency maps in Figure 5 not only for cases in which the learner classified the image correctly in both time steps, but also cases in which the learner classified the image correctly the first time and incorrectly the second time. Additionally, the previously mentioned evaluation steps, i.e., using a fixed attention map as baseline for the evaluation and evaluating the correlation between FSM and accuracy may be informative to illustrate the advantages of the proposed approach.
Clarity:
The paper is clearly written and easy to follow. One minor issue is that the first sentence of the third paragraph in Section 4 is not a full sentence and therefore difficult to understand.
Furthermore, on page 6, it is stated that the surrounding square $\hat{x}_i$ is 15 x 15 pixels, while the size of the square $x_i$ is 10 x 10. This appears strange, since it would mean that $x_i$ cannot be in the center of $\hat{x}_i$.


Generative Adversarial Networks (GANs) are a promising approach to language generation. The latest works introducing novel GAN models for language generation use n-gram based metrics for evaluation and only report single scores of the best run. In this paper, we argue that this often misrepresents the true picture and does not tell the full story, as GAN models can be extremely sensitive to the random initialization and small deviations from the best hyperparameter choice. In particular, we demonstrate that the previously used BLEU score is not sensitive to semantic deterioration of generated texts and propose alternative metrics that better capture the quality and diversity of the generated samples. We also conduct a set of experiments comparing a number of GAN models for text with a conventional Language Model (LM) and find that none of the considered models performs convincingly better than the LM.

This paper tackles the problem of evaluation of language generation models and particularly focuses on the comparison between GAN-based language models (GAN-LM) vs likelihood-based language models (MLE-LM). Studying the behaviour of current evaluation metrics for language generation as well as finding new ones is an important research topic. I believe that this paper makes a step in the right direction but the magnitude of that step may be insufficient for publication. I appreciate the efforts but I find most of the findings about BLEU not being a good metric and characteristic of reverse PPL rather unsurprising. The majority of the paper is dedicated to describing models / metrics which are well-known instead of performing more solid experimental evaluation (Results start at page 6). Instead, the authors could have focused more on the study of FD for language generation.
-- Details
-"Our main finding is that, when compared carefully, a conventional neural Language Model performs at least as well as any of the tested GAN models", however the authors don't compare with the recent MaskGAN model, which, according to (https://arxiv.org/abs/1801.07736) outperforms MLE variants.
- "We demonstrate that previously used n-gram matching, such as BLEU scores, is an insufficient metric": the fact that BLEU is not ideal for evaluation natural language generation has been pointed out in multiple related papers (e.g. https://arxiv.org/abs/1603.08023) and thus is not surprising.
- "We find that reporting results from the best single run or not performing sufficient tuning introduces significant bias in the reported results": as the authors point out in related works, the variance in GAN results which hinders meaningfulness of the reported results is a also a well-known problem (e.g. https://arxiv.org/pdf/1711.10337.pdf), therefore cannot be considered as a contribution.
- The observed behaviour (sensitivity to mode collapse, word swap, word removal) of the "reverse PPL" metric is pretty much expected, but I agree some experimental results are still interesting.
- On the contrary, I liked the study on the FD metric but I would have loved the paper to focus more on the study of the behaviour of that metric: for example, by examining the robustness under different base models, while the authors only test with the model by Conneau et. al, 2017.
- It would have been good to train a state-of-the-art language model architecture, e.g. AWD-LSTM, and to control regularization. I cannot see if the MLE-LM model is overfitting or not.
-- Style remarks:
- Moving the figures closer to the paragraph where they are described avoids the reader the burden of going back and forth through the paper.


We propose a novel adversarial learning framework in this work. Existing adversarial learning methods involve two separate networks, i.e., the structured prediction models and the discriminative models, in the training. The information captured by discriminative models complements that in the structured prediction models, but few existing researches have studied on utilizing such information to improve structured prediction models at the inference stage. In this work, we propose to refine the predictions of structured prediction models by effectively integrating discriminative models into the prediction. Discriminative models are treated as energy-based models. Similar to the adversarial learning, discriminative models are trained to estimate scores which measure the quality of predicted outputs, while structured prediction models are trained to predict contrastive outputs with maximal energy scores. In this way, the gradient vanishing problem is ameliorated, and thus we are able to perform inference by following the ascent gradient directions of discriminative models to refine structured prediction models. The proposed method is able to handle a range of tasks, e.g., multi-label classification and image segmentation. Empirical results on these two tasks validate the effectiveness of our learning method.

Building on the work of (Gygli et al., 2017), this paper introduces a training algorithm for energy-based models for structured prediction. Similar to Gygli et al. (2017), they train an energy-based discriminator, which matches the energy value of structured outputs with their target values assigned by a value function.
The authors present the learning algorithm in an adversarial learning framework by describing a structured prediction model G and a discriminator D. However, it is very confusing for me to understand the proposed formulation as an adversarial framework. In an adversarial framework, at the equilibrium, G could be used as a final prediction model, however, the predicted output of G are still low quality. For example, considering Table 1, the performance of G is exactly the same as NN baseline, which suggests that only the first term of Eq. 6  participates in the training of G (because L_g(G, y*) is the exact objective of the NN baseline).
What that I can easily relate to, however, is that this training algorithm is similar to Gygli et al. (2017), but uses G to get an initial point for gradient-based inference. We want this initial point to be close to the target value Eq. 6. We use the initial value (prediction of G) and the ground truth as the matching constraints (Eq. 5) (as well as the other samples that construct Eq. 10). This actually describes why D can refine the output of G (because it looks at it as an initial point that needs refinement), but the discriminators in the other adversarial frameworks can't refine that much (since they have reached the equilibrium). I would love to hear authors comments on my concern regarding the proposed adversarial framework.
Other comments:
1) Lg is a surrogate loss, not the task-loss. Task-loss could be F1, IOU, BLEU, etc, which is the ultimate performance measure on a task.
2) The authors refer to G as a structured prediction model but starting from Section 4, they have switched to call it a classifier, which is confusing.
3) "Gygli et al. (2017) found that the key to learning energy-based models is generating proper training data.":  Is this a general statement for every energy-based model? I understand its effect when matching values, but is it still true for other training algorithms such as structural SVM training (Belanger and McCallum, 2016)? Do you have evidence to support it?
4) "In the experiment, we adopt the fully convolutional network (FCN) (Long et al., 2015) baseline model proposed in (Gygli et al., 2017) as our segmentation network. It consists of three 5 × 5 convolutional layers and two deconvolution layer.": The text from Gygli et al. (2017) says three convolutional layers and two fully connected layers. Are you using the same architecture? If not, can you describe the architecture in more details?
5) The qualitative results for Gygli et al. (2017) appear in https://gyglim.github.io/deep-value-net/. The reported output for DVN row is significantly worse than the segmentation results of the same horses specifically for columns 4 and 8, while the overall reported IOU in Table 2 is exactly the same. Can you describe the source of this disagreement?
6) Is having a continuous domain for value function v essential for the proposed training algorithm?
=== After rebuttal ===
I am not convinced that the improved performance is because of the adversarial training. I trained a simple MLP and with the right amount of regularization it gets 42.0% f1 score on Bibtex, so I am not sure that the adversarial training is very essential here.


Importance sampling is a common approach to off-policy learning in reinforcement learning. While it is consistent and unbiased, it can result in high variance updates to the parameters for the value function. Weighted importance sampling (WIS) has been explored to reduce variance for off-policy policy evaluation, but only for linear value function approximation. In this work, we explore a resampling strategy to reduce variance, rather than a reweighting strategy. We propose Importance Resampling (IR) for off-policy learning, that resamples experience from the replay buffer and applies a standard on-policy update. The approach avoids using importance sampling ratios directly in the update, instead correcting the distribution over transitions before the update. We characterize the bias and consistency of the our estimator, particularly compared to WIS. We then demonstrate in several toy domains that IR has improved sample efficiency and parameter sensitivity, as compared to several baseline WIS estimators and to IS. We conclude with a demonstration showing IR improves over IS for learning a value function from images in a racing car simulator.

This paper introduces the concept of Sampling Importance Resampling (SIR) and give a simple method to adjust the off-policyness in the TD update rule of (general) value function learning, as an alternative of importance sampling. The authors argue that this resampling technique has several advantages over IS, especially on the stability with respect to step-size if we are doing optimization based the reweighted/resampled samples. In experiment section they show the sensitivity to learning rate of IR TD learning is closer to the on-policy TD learning, comparing with using IS or WIS.
Main comments:
The proposed IR technique is simple and definitely interesting in RL settings. The advantage about sensitivity of step-size choice in optimization algorithm looks appealing to me, since that is a very common practical issue with IS weighted objective. However I feel both the theoretical analysis and empirical results will be more convinced to me if a more complete analysis is presented. Especially considering that the importance resampling itself is well known in another field, in my point of view, the main contribution/duty of this paper would be introducing it to RL, comparing the pros/cons with popular OPPE methods in RL, and characterize what is the best suitable scenario for this method. I think the paper could potentially do a better job. See detailed comments:
1. The assumption of Thm 3.2 in main body looks a little bit unnatural to me. Why can we assume that the variance is bounded instead of prove what is the upper bound of variance in terms of MDP parameters? I believe there exists an upper bound so that result would be correct, but I’m just saying that this should be part of the proof to make the theorem to be complete.
2. If my understanding to section 3.3 is correct, the variance of IR here is variance of IR just for one minibatch. Then this variance analysis also seems a little bit weird to me. Since IR/IR-BC is computed online (actually in minibatch), I think a more fair comparison with IS/WIS might be giving them the same number of computations over samples. E.g. I would like to see the result of averaged IR/IR-BC estimator (over n/k minibatch’s) in either slicing window (changed every time) or fully offline buffer, where n is the number of samples used in IS/WIS and k the size of minibatch. I think it would be more informative than just viewing WIS as an (upper bound) benchmark since it uses more samples than.
3. From a higher level, this paper considers the problem of learning policy-value function with off-policy data. I think in addition to TD learning with IS adjustment, fitted Q iteration might be a natural baseline to compare with. It is also pretty widely-used and simple. Unlink TD, FQI does not need off-policy adjustment since it learns values for each action. I think that can be a fair and necessary baseline to compare to, at least in experiment section.
4. A relatively minor issue: I’m glad to see the author shows how sensitive each method is to the change of learning rate. I think it would be better to show some results to directly support the argument in introduction -- “the magnitude of the updates will vary less”, and maybe some more visualizable results on how stable the optimization is using IS and IR. I really think that is the most appealing point of IR to me.
Minor comments:
5. The authors suggest that the second part of Var(IR), stated in the fifth line from the bottom in page 5, is some variability not related to IS ratio but just about the update value it self. I think that seems not the case since the k samples (\delta_ij’s, j=1 to k) actually (heavily) depend on IS raios, unless I missed something here. E.g. in two extreme case where IS weights are all ones or IS weights are all zero except for one (s,a) in the buffer, the variance is very different and that is because of IS ratio but not the variance of updates themselves.
6. Similar with (5), in two variance expressions on the top of page 6, it would be better to point out that the distribution of k samples are actually different in two equations. One of them is sampled uniformly from buffer and the other is proportional to IS ratios.
7. I think it is a little bit confused to readers when sometimes both off-policy learning and off-policy policy evaluation are used to describe the same setting. I would personally prefer use off-policy (policy) learning only in the “control” setting: learning the optimal policy or the optimal value function, and use the term off-policy policy evaluation referring to estimating a given policy’s value function. Though I understand that sometimes we may say “learning a policy value function for a given policy”, I think it might be better to clarify the setting and later use the same term in the whole paper.
Overall, I think there are certainly some interesting points about the IR idea in this paper. However the issues above weakens my confidence about the clarity and completeness of the analysis (in both theory and experiment) in this paper.


Randomly initialized first-order optimization algorithms are the method of choice for solving many high-dimensional nonconvex problems in machine learning, yet general theoretical guarantees cannot rule out convergence to critical points of poor objective value. For some highly structured nonconvex problems however, the success of gradient descent can be understood by studying the geometry of the objective. We study one such problemcomplete orthogonal dictionary learning, and provide converge guarantees for randomly initialized gradient descent to the neighborhood of a global optimum. The resulting rates scale as low order polynomials in the dimension even though the objective possesses an exponential number of saddle points. This efficient convergence can be viewed as a consequence of negative curvature normal to the stable manifolds associated with saddle points, and we provide evidence that this feature is shared by other nonconvex problems of importance as well.

The authors analyze the convergence performance of Riemannian gradient descent based algorithm for the dictionary learning problem with orthogonal dictionary and sparse factors. They demonstrate a polynomial time convergence from a random initialization for a smooth surrogate objective for the original non-smooth one. The problem and the analysis are of interest, but I have several questions regarding the paper as follows.
My first concern is that the analysis is on a smooth surrogate of the non-smooth sparse minimization for solving the dictionary learning problem, so it is not clear what is relationship between the global minimizer of the smooth problem to the underlying true dictionary. More specifically, how far is the global minimizer of problem (1) or (2) to the true dictionary parameter, and whether they share (approximately) the space or components regarding the sparse factors. Without clarifying this, it is not well motivated why we are interested in studying the problem considered in this paper at the beginning. Intuitively, since the recovered factors are not sparse anymore, it will impact the dictionary accordingly due to the linear mapping, which may lead to a very different set of dictionary components. Thus, explicit explanation is necessary here to avoid such degenerate case.
My second concern is the eligibility of assuming the dictionary A to be an identity matrix and extending it to the general orthogonal matrix case. The analysis uses the fact that rows of A are canonical basis, i.e., each row only has one non-zero entry. I do not see a straightforward extension by replacing A to be an orthogonal matrix as the authors claimed on page 3, since then the inner product of one row of A and one column of X is not just the corresponding entry of the column of X. It will be helpful if the authors can explain this explicitly or adjust the analysis accordingly to make this valid.
Another issue is the clarity of the paper. Some statements in the paper are not very clear. Form example, on page 3, third paragraph of Section 3, what does row(Y) = row(X_0) mean? Also, in eqn (1), y_k means k-th column of Y, and in eqn (2), q_i means i-th entry of q? Since both are bold lower case letters, clear distinction will help. Moreover, the reference use (. ) instead [ .], which can be confusing sometimes.


AD V-BNN: IMPROVED ADVERSARIAL DEFENSE THROUGH ROBUST BAYESIAN NEURAL NETWORK
We present a new algorithm to train a robust neural network against adversarial attacks. Our algorithm is motivated by the following two ideas. First, although recent work has demonstrated that fusing randomness can improve the robustness of neural networks (Liu et al., 2017), we noticed that adding noise blindly to all the layers is not the optimal way to incorporate randomness. Instead, we model randomness under the framework of Bayesian Neural Network (BNN) to formally learn the posterior distribution of models in a scalable way. Second, we formulate the mini-max problem in BNN to learn the best model distribution under adversarial attacks, leading to an adversarial-trained Bayesian neural network. Experiment results demonstrate that the proposed algorithm achieves state-of-the-art performance under strong attacks. On CIFAR-10 with VGG network, our model leads to 14% accuracy improvement compared with adversarial training (Madry et al., 2017) and random self-ensemble (Liu et al., 2017) under PGD attack with 0.035 distortion, and the gap becomes even larger on a subset of ImageNet 1 .

After feedback: I would like to thank the authors for careful revision of the paper and answering and addressing most of my concerns. From the initial submission my main concern was clarity and now the paper looks much more clearer.
I believe this is a strong paper and it represents an interesting contribution for the community.
Still things to fix:
a) a dataset used in 4.2 is not stated
b) missing articles, for example, p.5 ".In practice, however, we need a weaker regularization for A small dataset or A large model"
c) upper case at the beginning of a sentence after question: p.8 "Is our Adv-BNN model susceptible to transfer attack? we answer" - "we" -> "We"
====================================================================================
The paper proposes a Bayesian neural network with adversarial training as an approach for defence against adversarial attacks.
Main pro:
It is an interesting and reasonable idea for defence against adversarial attacks to combine adversarial training and randomness in a NN (bringing randomness into a new level in the form of a BNN), which is shown to outperform both adversarial training and random NN alone.
Main con:
Clarity. The paper does not crucially lack clarity but some claims, general organisation of the paper and style of quite a few sentences can be largely improved.
In general, the paper is sound,  the main idea appears to be novel and the paper addresses the very important and relevant problem in deep learning such as defence against adversarial attacks. Writing and general presentation can be improved especially regarding Bayesian neural networks, where some clarity issues almost become quality issues. Style of some sentences can be tuned to more formal.
In details:
1. The organisation of Section 1.1 can be improved: a general concept "Attack" and specific example "PGD Attack" are on the same level of representation, while it seems more logical that "PGD Attack" should be a subsection of "Attack". And while there is a paragraph "Attack" there is no paragraph "Defence" but rather only specific examples
2. The claim “we can either sample w ∼ p(w|x, y) efficiently without knowing the closed-form formula through the method known as Stochastic Gradient Langevin Dynamics (SGLD) (Welling & Teh, 2011)” sounds like SGLD is the only sampling method for BNN, which is not true, see, e.g., Hamiltonian Monte Carlo (Neal’s PhD thesis 1994). It is better to be formulated as "through, for example, the method ..."
3. Issues regarding eq. (7):
   a) Why there is an expectation over (x, y)? There should be the joint probability of all (x, y) in the evidence.
   b) Could the authors add more details about why it is the ELBO given that it is unconventional with adversarial examples added?
   c)  It seems that it should be log p(y | x^{adv}, \omega) rather than p(x^{adv}, y | \omega).
   d) If the authors assume noise component, i.e., y = f(x; \omega) + \epsilon, then they do not need to have a compulsory Softmax layer in their network, which is important, for example, for regression models. Then the claim “our Adv-BNN method trains an arbitrary Bayesian neural network” would be more justified
4. It would make the paper more self-contained if the Bayes by Backprop algorithm would be described in more details (space can be taken from the BNN introduction). And it seems to be a typo that it is Bayes by Backprop rather than Bayes by Prop
5. There are missing citations in the text:
    a) no models from NIPS 2017 Adversarial Attack and Defence competition (Kurakin et al. 2018) are mentioned
    b) citation to justify the claim “C&W attack and PGD attack (mentioned below) have been recognized as
two state-of-the-art white-box attacks for image classification task”
    c) “we can approximate the true posterior p(w|x, y) by a parametric distribution q_θ(w), where the unknown parameter θ is estimated by minimizing KL(q_θ(w) || p(w|x, y)) over θ” - there are a variety of works in approximate inference in BNN, it would be better to cite some of them here
    d) citation to justify the claim "although in these cases the KL-divergence of prior and posterior is hard to compute and practically we replace it with the Monte-Carlo estimator, which has higher variance, resulting in slower convergence rate.”
6. The goal and result interpretation of the correlation experiment is not very clear
7. From the presentation of Figure 4 it is unclear that this is a distribution of standard deviations of approximated posterior.
8. “To sum up, our Adv-BNN method trains an arbitrary Bayesian neural network with the adversarial examples of the same model” – unclear which same model is meant
9. "Among them, there are two lines of work showing effective results on medium-sized convolutional networks (e.g., CIFAR-10)" - from this sentence it looks like CIFAR-10 is a network rather than a dataset
10. In "Notations" y introduction is missing
11. It is better to use other symbol for perturbation rather than \boldsymbol\delta since \delta is already used for the Dirac delta function
12. “via tuning the coefficient c in the composite loss function” – the coefficient c is never introduced
Minor:
1. There are a few missing articles, for example, in Notations, “In this paper, we focus on the attack under THE norm constraint…”
2. Kurakin et al. (2017) is described in the past tense whereas Carlini & Wagner (2017a) is described in the present tense
3. Inner brackets in eq. (2) are bigger than outer brackets
4. In eq. (11) $\delta$ is not bold
5. In eq. (12) it seems that the second and third terms should have “-” rather than “+”
6. Footnote in page 6 seems to be incorrectly labelled as 1 instead of 2


Much of contemporary sensorimotor learning assumes that one is already given a complex agent (e.g., a robotic arm) and the goal is to learn to control it. In contrast, this paper investigates a modular co-evolution strategy: a collection of primitive agents learns to self-assemble into increasingly complex collectives in order to solve control tasks. Each primitive agent consists of a limb and a neural controller. Limbs may choose to link up to form collectives, with linking being treated as a dynamic action. When two limbs link, a joint is added between them, actuated by the 'parent' limb's controller. This forms a new 'single' agent, which may further link with other agents. In this way, complex morphologies can emerge, controlled by a policy whose architecture is in explicit correspondence with the morphology. In experiments, we demonstrate that agents with these modular and dynamic topologies generalize better to test-time environments compared to static and monolithic baselines. Project videos are available at https:// doubleblindICLR19.github.io/self-assembly/.

Summary:
--------------
The paper considers the problem of constructing compositional robotic morphologies that can solve different continuous control tasks in a (multi-agent) reinforcement learning setting. The authors created an environment where the actor consists of a number of primitive components which interface with each other via "linking" and construct a morphology of a robot. To learn in such an environment, the authors proposed a graph neural network policy architecture and showed that it is better than the baselines on the proposed tasks.
I find the idea of learning in environments with modular morphologies as well as the proposed tasks interesting. However, the major drawback of the paper is the lack of any reasonable details on the methods and experiments. It's hard to comment on the novelty of the architecture or the soundness of the method when such details are simply unavailable.
More comments and questions are below. I would not recommend publishing the paper in the current form.
Comments:
----------------
- If I understand it correctly, each component ("limb") represents an agent. Can you define precisely (ie mathematically) what the observations and actions of each agent are?
- Page 4, paragraph 2: in the inline equation, you write that a sum over actions equals policy applied to a sum over states. What does it mean? My understanding of monolithic agents is that observations and actions must be stacked together. Otherwise, the information would be lost.
- Page 4, paragraphs 3-(end of section): if I understand it correctly, the proposed method looks similar to the problem of "learning to communicate" in a cooperative multi-agent setting. This raises the question, how exactly the proposed architecture is trained? Is it joint learning and joint execution (ie there's a shared policy network, observation and action spaces are shared, etc), or not? All the details on how to apply RL to the proposed setup are completely omitted.
- Is the topology of the sub-agents restricted to a tree? Why so? How is it selected (in cases when it is not hand-specified)?
- From the videos, it looks like certain behaviors are very unphysical or unrealistic (eg parts jumping around and linking to each other). I'm wondering which kind of simulator was used? How was linking defined (on the simulator level)? It would be nice if such environments with modular morphologies were built using the standard simulators, such as MuJoCo, Bullet, etc.
All in all, despite potentially interesting ideas and setup, the paper is sloppily written, has mistakes, and lacks crucial details.


This work examines batch reinforcement learning-the task of maximally exploiting a given batch of off-policy data, without further data collection. We demonstrate that due to errors introduced by extrapolation, standard off-policy deep reinforcement learning algorithms, such as DQN and DDPG, are only capable of learning with data correlated to their current policy, making them ineffective for most off-policy applications. We introduce a novel class of off-policy algorithms, batch-constrained reinforcement learning, which restricts the action space to force the agent towards behaving on-policy with respect to a subset of the given data. We extend this notion to deep reinforcement learning, and to the best of our knowledge, present the first continuous control deep reinforcement learning algorithm which can learn effectively from uncorrelated off-policy data.

Summary:
Proposes BCRL for learning from a fixed collection of off-policy experience (I'll call this the "training data"). BCRL attempts to avoid backing up values from states that are not present in the training data, on the assumption that the current estimates of these values are likely to be inaccurate. In the continuous state-action case, this is accomplished by training a generative model to propose, given a state `s`, an action `a` such that a transition similar to `(s, a)` is in the training data. A secondary policy is then trained to perturb the proposed action within a constrained region to maximize value. BCRL outperforms DDPG and DQN when learning from fixed data, but BCRL is slightly worse than behavior cloning at learning to reproduce an expert policy that does not take exploration actions.
Review:
The overall approach is sound. The problem of extrapolation is intuitively obvious, but not something I had thought about before. I think typically exploration would correct the problem since states with over-estimated values would become more likely to be reached, giving an opportunity to get a better estimate.
The learning setting is closer to imitation learning than to what I would call RL, since the BCRL approach essentially avoids extrapolation error by ignoring the parts of the problem that are not represented in the training data. The well-known problem with behavior cloning is compounding errors once the agent strays into areas of the state space that are far from the training data. To me "off-policy RL" implies that the goal is to learn a complete policy from off-policy data. I think the "competitors" to which BCRL should be compared are imitation learning algorithms address noisy demonstrations, and not so much off-policy RL algorithms. It would also be interesting to see the generalization performance of BCRL outside of its training data.
The BCRL idea might be applicable in a conventional RL setting as well, since the initial stages of learning could be subject to a similar extrapolation error until there has been enough exploration. A comparison to something like TRPO in this setting would be interesting.
The paper is well-written with good coverage of related literature. There are a few points where the technical content is imprecise, which I note below.
Comments / Questions:
* Could one obtain a similar effect to BCRL by simply initializing the value estimates pessimistically?
* Sec 4.1: Since B is a set of (s, a, s', r) tuples, what does it mean for a state s' to be "in B"? Similar question for state-action tuples (s, a).
* As you note in the appendix, the construction in Sec 4.1 is essentially creating a new MDP that contains only the transitions that occur in the training data. I'd suggest stating as much in the main paper for intuition.
* Sec 4.2 / 5: The perturbation constraint \Phi is set to 0.05 in the experiments. Since the actions in these control problems are vectors, what does a scalar constraint correspond to? How is the constraint enforced during learning?
* What are the distance functions D_S and D_A?
Pros:
* A good approach to applying RL methods in the "imitation-like" setting. I've seen similar things attempted before, but this method makes more sense.
Cons:
* The learning setting is more like "fuzzy" behavior cloning from noisy data than off-policy RL. Experimental comparison against more-sophisticated imitation learning approaches is missing.


ADAPTIVITY OF DEEP RELU NETWORK FOR LEARN- ING IN BESOV AND MIXED SMOOTH BESOV SPACES: OPTIMAL RATE AND CURSE OF DIMENSIONALITY
Deep learning has shown high performances in various types of tasks from visual recognition to natural language processing, which indicates superior flexibility and adaptivity of deep learning. To understand this phenomenon theoretically, we develop a new approximation and estimation error analysis of deep learning with the ReLU activation for functions in a Besov space and its variant with mixed smoothness. The Besov space is a considerably general function space including the Hölder space and Sobolev space, and especially can capture spatial inhomogeneity of smoothness. Throughout the analysis in the Besov space, it is shown that deep learning can achieve the minimax optimal rate and outperform any nonadaptive (linear) estimator such as kernel ridge regression, which shows that deep learning has higher adaptivity to the spatial inhomogeneity of the target function than other estimators such as linear ones. In addition to this, it is shown that deep learning can avoid the curse of dimensionality if the target function is in a mixed smooth Besov space. We also show that the dependency of the convergence rate on the dimensionality is tight due to its minimax optimality. These results support high adaptivity of deep learning and its superior ability as a feature extractor.

This paper describes approximation and estimation error bounds for functions in Besov spaces using estimators corresponding to deep ReLU networks. The general idea of connecting network parameters such as depth, width, and sparsity to classical function spaces is interesting and could lead to novel insights into how and why these networks work and under what settings. The authors carefully define Besov spaces and related literature, and overall the paper is clearly written.
Despite these strengths, I'm left with several questions about the results. The most critical is this: piecewise polynomials are members of the Besov spaces of interest, and ReLU networks produce piecewise linear functions. How can piecewise linear approximations of piecewise polynomial functions lead to minimax optimal rates? The authors' analysis is based on cardinal B-spline approximations, which generally makes sense, but it seems like you would need more terms in a superposition of B-splines of order 2 (piecewise linear) than higher orders to approximate a piecewise polynomial to within a given accuracy. The larger number of terms should lead to worse estimation errors, which is contrary to the main result of the paper. I don't see how to reconcile these ideas.
A second question is about the context of some broad claims, such as that the rates achieved by neural networks cannot be attained by any linear or nonadaptive method. Regarding linear methods, I agree with the author, but I feel like this aspect is given undue emphasis. The key paper cited for rates for linear methods is the Donoho and Johnstone Wavelet Shrinkage paper, in which they clearly show that nonlinear, nonadaptive wavelet shrinkage estimators do indeed achieve minimax rates (within a log factor) for Besov spaces. Given this, how should I interpret claims like "any linear/non-linear approximator
with fixed N -bases does not achieve the approximation error ... in some parameter settings such as 0 < p < 2 < r "?
Wavelets provide a fixed N-basis and achieve optimal rates for Besov spaces. Is the constraint on p and r a setting in which wavelet optimality breaks down? If not, then I don't think the claim is correct. If so, then it would be helpful to understand how relevant this regime for p and r is to practical settings (as opposed to being an edge case).
The work on mixed Besov spaces (e.g. tensor product space of 1-d Besov spaces) is a fine result but not surprising.
A minor note: some of the references are strange, like citing a 2015 paper for minimax rates for Besov spaces that have been known for far longer or a 2003 paper that describes interpolation spaces that were beautifully described in DeVore '98. It would be appropriate to cite these earlier sources.


In this paper we propose autoencoder architectures for learning a Cohen-Welling (CW)-basis for images and their rotations. We use the learned CW-basis to build a rotation equivariant classifier to classify images. The autoencoder and classifier architectures use only tensor product nonlinearity. The model proposed by Cohen & Welling (2014) uses ideas from group representation theory, and extracts a basis exposing irreducible representations for images and their rotations. We give several architectures to learn CW-bases including a novel coupling AE architecture to learn a coupled CW-bases for images in different scales simultaneously. Our use of tensor product nonlinearity is inspired from recent work of Kondor (2018a). Our classifier has very good accuracy and we use fewer parameters. Even when the sample complexity to learn a good CW-basis is low we learn classifiers which perform impressively. We show that a coupled CW-bases in one scale can be deployed to classify images in a classifier trained and tested on images in a different scale with only a marginal dip in performance.

Recently there has been a spate of work on generalized CNNs that are equivariant to various symmetry groups, such a 2D and 3D rotations, the corresponding Euclidean groups (comprising not just rotations but also translations) and so on. The approach taken in most of the recent papers is to explicitly build in these equivariances by using the appropriate generalization of convolution. In the case of nontrivial groups this effectively means working in Fourier space, i.e., transforming to a basis that is adapted to the group action. This requires some considerations from represntation theory.
Earlier, however, there was some less recognized work by Cohen and Welling on actually learning the correct basis itself from data. The present paper takes this second approach, and shows for a simple task like rotated MNIST, the basis can be learned from a remarkably small amount of data, and actually performs even better than some of the fixed basis methods. There is one major caveat: the nonlinearity itself has to be rotation-covariant, and for this purpose they use the recently introduced tensor product nonlinearities.
The paper is a little rough around the edges. In the first 4 pages it launches into an exposition of ideas from representation theory which is too general for the purpose: SO(2) is a really simple commutative group, so the way that "tensor product" representations reduce to irreducibles could be summed up in the formula  $e^{-2\pi i k_1 x}e^{-2\pi i k_2 x}=e^{-2\pi i (k_1+k_2) x}$. I am not sure why the authors choose to use real representations (maybe because complex numbers are not supported in PyTorch, but this could easily be hacked) and I find that the real representations make things unnecessarily complicated. I suspect that at the end of the day the algorithm does something very simple (please clarify if working with
real representations is somehow crucial).
But this is exactly the beauty of the approach. The whole algorithm is very rough, there are only two layers (!), no effort to carefully implement nice exact group convolutions, and still the network is as good as the competition. Another significant point is that this network is only equivariant to rotations and not translations.
Naturally, the question arises why one would want to learn the group adapted basis, when one could just compute it explicitly. There are two interesting lessons here that the authors could emphasize more:
1. Having a covariant nonlinearity is strong enough of a condition to force the network to learn a group adapted (Cohen-Welling) basis. This is interesting because Fourier space ("tensor") nonlinearities are a relatively new idea in the literature. This finding suggests that the nonlinearity might actually be more important than the basis.
2. The images that the authors work on are not functions on R^2, but just on a 28x28 grid. Rotating a rasterized image with eg. scikit-rotate introduces various artifacts. Similarly, going back and forth between a rasterized and polar coordinate based representation (which is effectively what would be required for "Harmonic Networks" and other Fourier methods) introduces messy interpolation issues. Not to mention downsampling, which is actually addressed in the paper. If a network can figure out how to best handle these issues from data, that makes things easier.
The experiments are admittedly very small scale, although some of the other publications in this field also only have small experiments. At the very least it would be nice to have standard deviations on the results and some measure of statistical significance. It would be even nicer to have some visualization of the learned bases/filters, and a bare bones matrix-level very simple description of the algorith. Again, what is impressive here is that such a small network can learn to do this task reasonably well.
Suggestions:
1. Also cite the Tensor Field Networks of Thomas et al in the context of tensor product nonlinearities.
2. Clean up the formatting. "This leads us to the following" in a line by itself looks strange. Similarly "Classification ising the learned CW-basis". I think something went wrong with \itemize in Section 3.1.


Graph networks have recently attracted considerable interest, and in particular in the context of semi-supervised learning. These methods typically work by generating node representations that are propagated throughout a given weighted graph. Here we argue that for semi-supervised learning, it is more natural to consider propagating labels in the graph instead. Towards this end, we propose a differentiable neural version of the classic Label Propagation (LP) algorithm. This formulation can be used for learning edge weights, unlike other methods where weights are set heuristically. Starting from a layer implementing a single iteration of LP, we proceed by adding several important non-linear steps that significantly enhance the label-propagating mechanism. Experiments in two distinct settings demonstrate the utility of our approach.

**** After Revision ***********
I thank the authors for diligently revising the paper according to the reviewers' suggestions. I have increased my score for the paper. I still think the experimental evaluation can be more thorough. For example, it would be good to show the effect of varying the \tau parameter and the number of available labels (k). It would also be good to experiment with the Flickr graph without any sparsification and to add uncertainty estimates to the results in Table 1.
**** After Revision ***********
This paper proposes a framework for non-linear label propagation where the weights are learned simultaneously. There are model specific and experimental setup design decisions that require justification. There also needs to be a number of ablation studies to justify the effectiveness of the different components of this framework.  Finally, there seems to be an insufficient comparison (both experimentally and theoretically) to the large amount of related literature.
- What is the total number of parameters in the proposed network? Please clarify how this is "relatively few parameters" as compared to other methods.
- Please compare how your method for learning weights relates to the following papers and the references therein [1,2]
[1] Online Learning of Multiple Tasks and Their Relationships. Saha et al, AISTATS, 2011.
[2] Convex Learning of Multiple Tasks and their Structure, Cilberto et al, 2015.
- It would be good to have an ablation study in order to discern what is the contribution of learning the weights vs propagating labels (instead of embeddings).
- For clarity, please specify that \theta are the parameters to be learned.
- Please explain the intuition of using entropy and KL divergence for the attention weights. Shouldn't the attention for an edge be inversely proportional to the entropy i.e. the attention should be higher if the neighboring node's label is more certain?
- Instead of the bifurcation mechanism proposed in section 3.2, isn't it possible to use a threshold to round the resulting prediction to a hard label?
- In equation 13, are the hyper-parameters a, b tuned using cross-validation? Can't we learn the \tau in the same training procedure? Please justify this design decision?
- What is the performance if the loss in equation 14 is replaced by the standard empirical loss? There needs to be an ablation study on this.
- If the node features are available, how are they used in this framework?
- In the experimental section, why is k chosen to be equal to 1%? Please show results while varying this.
- Please justify the line "parameterizes w using a small number (~20) of informative features based on the raw features (e.g., dimensionality reduction), the graph (e.g., edge betweenness), and the labeled set (e.g., distance from labeled nodes). " Isn't it possible to get similar performance by reducing the number of parameters so that model doesn't overfit?
- Please clearly state what is the difference in the framework from the Kipf and Welling, 2016 paper?
- Why isn't there a comparison to methods like Graph-Sage?
- Please explain this line "LPNnobif degrades with large T, and even \tau slightly above 1 makes a difference"
- Finally, please explain the trend in the results in Table 1. For example, why is the performance of the proposed method poor on the Flickr dataset, but better on the DBLP dataset?
- It would good to have uncertainty estimates for the results reported in Table 1.