Balancing exploration and exploitation remains a key challenge in reinforcement learning (RL). State-of-the-art RL algorithms suffer from high sample complexity, particularly in the sparse reward case, where they can do no better than to explore in all directions until the first positive rewards are found. To mitigate this, we propose Rapidly Randomly-exploring Reinforcement Learning (R3L). We formulate exploration as a search problem and leverage widely-used planning algorithms such as Rapidly-exploring Random Tree (RRT) to find initial solutions. These solutions are used as demonstrations to initialize a policy, then refined by a generic RL algorithm, leading to faster and more stable convergence. We provide theoretical guarantees of R3L exploration finding successful solutions, as well as bounds for its sampling complexity. We experimentally demonstrate the method outperforms classic and intrinsic exploration techniques, requiring only a fraction of exploration samples and achieving better asymptotic performance.

This paper proposes a method, R3L, for exploration in reinforcement learning. R3L performs an exploration procedure before policy optimization. In R3L exploration, it considers the task as a planning problem, and applies RRT to find feasible solutions (trajectories). Then it applies a warm start procedure for policy optimization, where the feasible solutions found by RRT are used to initialize the policy by supervised learning. The paper provides theoretical guarantees of R3L exploration finding feasible solutions. Empirically, the algorithmic contribution is demonstrated by comparing with information theoretical exploration methods in benchmark control domains.
The paper makes two strong assumptions, which are made to guarantee RRT can be successfully applied in the task. However, it is unlikely that these two assumptions can hold in RL problems we consider in general, where the learning agent only have access to the transition data gathered by interactions with the environment. This makes the proposed R3L algorithm not a general method for exploration in RL. Due to this reason I think this paper should be rejected.
The first assumption is that random states can be uniformly sampled from the MDP state space. The author further argue that sampling a random state is typically equivalent to trivially sampling a hyper-rectangle. But isn't this a domain-dependent property? For example, how to sample a random state in Atari games and decide if it is valid? By the method proposed in the paper, one need to sample a random image, then apply a function to decide if the image is valid in the game. How to get such a function? The second assumption is that the environment state can be set to an arbitrary state. This basically assumes the learning agent is available to the transition function, so that a new state can be added to the current search tree. But again, this assumption might not be appropriate in the general RL framework.
For the theoretical contribution, the paper shows that RRT is complete with high probability, which is a standard result of RRT. In experiments, R3L is compared with VIME. But is this a fair comparison since R3L assumes to have a generative model? The tested domains are picked such that RRT can be directly applied. Can R3L be applied in domains like mujoco or Atari games?
The main idea of this paper is to deal with exploration using planning algorithms. But once a generative model of the environment is given, the exploration problem will be very different with the exploration considered in RL. I would like to improve my score if the author can demonstrate the efficiency of R3L with a learned generative model.
Minor issues:
1. Can you give more details about how pi_l is learned in Algorithm 1? In line 9 an action is generated using pi_l(s_near, s_rand-s_near). But in line 11, the action is again updated by ({s_near, s_rand-s_near}, a), which is very confusing.
2. The notations for state and valid state are very confusing. In 3.1, the transition and reward functions are defined on S. But later in the paper, S contains states that are not valid. What's the transitions and rewards on invalid states?


Recent research has shown that CNNs are often overly sensitive to high-frequency textural patterns. Inspired by the intuition that humans tend to be more sensitive to lower-frequency (larger-scale) patterns, we design a regularization scheme that penalizes large differences between adjacent components within each convolutional kernel. We apply our regularization onto several popular training methods, demonstrating that the models with the proposed smooth kernels enjoy improved adversarial robustness. Further, building on recent work establishing connections between adversarial robustness and interpretability, we show that our method appears to give more perceptually-aligned gradients.

Paper summary:  This paper argues that reducing the reliance of neural networks on high-frequency components of images could help robustness against adversarial examples. To attain this goal, the authors propose a new regularization scheme that encourages convolutional kernels to be smoother. The authors augment standard loss functions with the proposed regularization scheme and study the effect on adversarial robustness, as well as perceptual-alignment of model gradients.
Comments: I will first discuss some high-level concerns I have with the paper, followed by more specific comments.
Motivation and general idea:
The idea of improving model robustness by eliminating high-frequency components in the data is not new. This has in fact been the motivation behind several (later shown to be unsuccessful) defenses such as JPEG compression. In recent work, Yin et al. explore this phenomenon in depth, and importantly discuss how adversarial examples are by no means entirely a high-frequency phenomenon. Specifically, they show that while adversarial examples for natural models do tend to be biased towards higher-frequency components, this is not true for robust models. They argue that it is in general always possible to find adversarial examples in the lower end of the frequency spectrum (for instance, transfer attacks from robust models). Thus, based on evidence from prior work, it is unlikely that a defense based entirely on removing high-frequency components from the input data can be successful.
Empirical evaluation:
There are significant issues with the empirical evaluation of the proposed defense. In particular, in Table 1 (also Figures 7-9 in the Appendix):
1. The numbers suggest that FGSM is more successful as an attack than PGD. The later is a multi-step version of the former, and hence should be strictly better (more successful in lowering model accuracy). This clearly highlights that the PGD attack is not being used correctly/not run with enough steps, and thus the numbers are not an accurate reflection of model robustness.
2. At a higher-level, the numbers that the authors highlight in the table are the best performance over attacks. This is not the correct way to evaluate robustness, which must always be reported as the *worst-case performance* of the model and hence the lowest accuracy over attacks. If one takes this into consideration, it is clear that the proposed regularization does not really improve robustness.
3. Furthermore, the authors state they use default parameters from Foolbox to evaluate their models. The issue with this is evident in the MNIST/FashionMNIST results where even with arbitrary eps, strong attacks like PGD are not able to break the model. This does not mean that the model is robust, it just means that the default hyperparameters from Foolbox cannot break the model. The authors need to re-run the attacks with different steps sizes/step counts for various attacks. The goal in evaluating model robustness should not be finding one attack (or one set of hyperparameters) which is not able to fool the model, but to show that *no attack* (for the given perturbation set) can lower model accuracy.
4. The eps that are used for the CIFAR-10 and ImageNet experiments are extremely large, and not standard in the literature. In fact, based on my experience, it is probably not possible to be (too) robust to eps as large as 0.1 on CIFAR as this is large enough to visibly change the class even for a human (cf. Tsipras et al.).
Thus, I think the robustness evaluation in Table 1 is incorrect and violates some basic sanity checks (such as PGD being stronger than FGSM), and thus does not accurately reflect the model’s true performance.
Other comments:
i. It is unclear why the sensitivity of the proposed regularization scheme to the scale of w can be fixed by subtracting the norm of w (the exp loss will just be raised to the power alpha if you scale weights by alpha). Dividing by the norm would be the more correct way to fix the scale invariance and the authors should include results with this regularization, at least in the Appendix.
ii. In Figure 1, the kernels and activations that are visualized are after the first convolutional layer. The authors should perform a similar visualization after the second layer (or later layers in general) because it is somewhat obvious that you will get these properties at the first layer with the proposed regularization scheme (if the lambda is correctly tuned). The part that is unclear is how effective the regularization is over repeated applications of convolution coupled with non-linearities such as ReLU/pooling.
iii. While the visualizations in Figures 2-6 are interesting, it seems that many of the cases in which +R looks semantically meaningful, it also looks meaningful with the base loss itself. It thus is unclear whether the perceptual alignment is coming from the base loss, or from the added regularization. Moreover, the perceptual alignment of models should improve when their dependance on “human-meaningless” features reduces. Since high-frequency components are likely “human-meaningless” features, it is plausible that the proposed regularization scheme makes activation maximization/gradients more perceptually alignment. However, I think this is somewhat orthogonal to (and is not enough to tell us anything about) the robustness of the model itself, which is affected by its reliance on many kinds of “non-robust” features (Ilyas et al., 2019) of which high-frequency components might just be one example.
iv. In recent work by Wang et al., the authors conduct an interesting experiment to see how dependent the prediction of a given classifier is to high-frequency components in the data (vs low frequency components) (cf. Figure 1 in their paper). It would be interesting to see this experiment replicated with the proposed regularization scheme.
Overall, I think there are significant issues with the paper, especially in the empirical evaluation section. The authors need to re-evaluate their models with the strongest possible form of the attack and demonstrate an improvement in worst-case performance to actually establish the merits of the proposed regularization scheme. Thus, I recommend rejection.
References:
Yin, Dong, et al. "A fourier perspective on model robustness in computer vision." arXiv preprint arXiv:1906.08988 (2019).
Tsipras, Dimitris, et al. "Robustness may be at odds with accuracy." arXiv preprint arXiv:1805.12152 (2018).
Ilyas, Andrew, et al. "Adversarial examples are not bugs, they are features." arXiv preprint arXiv:1905.02175 (2019).
Wang, Haohan, et al. "High Frequency Component Helps Explain the Generalization of Convolutional Neural Networks." arXiv preprint arXiv:1905.13545 (2019).


Training neural networks on image datasets generally require extensive experimentation to find the optimal learning rate regime. Especially, for the cases of adversarial training or for training a newly synthesized model, one would not know the best learning rate regime beforehand. We propose an automated algorithm for determining the learning rate trajectory, that works across datasets and models for both natural and adversarial training, without requiring any dataset/model specific tuning. It is a stand-alone, parameterless, adaptive approach with no computational overhead. We theoretically discuss the algorithm's convergence behavior. We empirically validate our algorithm extensively. Our results show that our proposed approach consistently achieves top-level accuracy compared to SOTA baselines in the literature in natural as well as adversarial training.

The paper considers the problem of automated adaptation of learning rate during (deep) neural network training. The use cases described are standard and adversarial training for image classification. Given the wide use of DNNs in computer vision (and other areas), learning rate tuning is clearly an important problem and is being actively researched.
The proposed learning rate adaptation procedure consists of a straightforward combination of learning rate halving/doubling and model checkpointing. Experimental results from implementing the adaptive learning rule for standard and adversarial training on CIFAR are provided. Multiple architectures are tested in each setting. The paper claims a primary advantage of the proposed learning rule to be that it requires no tuning as opposed to other rules such as SGD, Adam.
My decision is to reject the paper due to methodological issues with the experiments and lack of evidence wrt/ dataset variety. The paper should be considered a work-in-progress that may have potential in a more focused setting, e.g., adversarial training as described in the paper.
***
The major claim of the proposed algorithm not requiring any manual tuning is technically true but misleading. The algorithm does have parameters (SGD momentum, batch size, initial learning rate, patience) with values that were set somehow. In fact, a major methodological issue with the experiments is that the reader does not know if the datasets were used to both set these values and to assess performance, i.e., there are no obvious "held-out" datasets. Also, there is no rigorous or even informal justification of the settings. It could be that the paper is arguing that the specific values will result in competitive, if not better, performance than baselines across a variety of datasets - unfortunately, only two datasets are utilized in the experiments, and one, CIFAR10, is not considered challenging. This leads to the second issue with the paper: the experimental validation is not extensive wrt/ datasets which is significant given that the form of the evidence for the proposed method is almost entirely empirical.
Additionally, I don't agree that competitor algorithms should not be tuned b/c the proposed method does not require tuning. Even if the proposed method does not require tuning (as stated previously, I don't believe this to be accurate), that does not imply a fair comparison precludes tuning competitors via, e.g., cross validation. The only relevant quantities are final test-set performance and total training time/resources required.
The well-known interdependence between learning rate and batchsize as noted in e.g., Hoffer et al. (2018), is not addressed by the experiments. Batchsizes in the experiments vary, but no justification is provided for how these are selected.
Finally, the paper is unfinished as some experimental runs were not complete at the time of submission.
On the positive side, the general point about the necessity of learning rate tuning for adversarial training (described in the fourth paragraph of the introduction) is a very good one, and there may be an opportunity for a more focused application of the proposed algorithm perhaps among further datasets and considering additional, alternative attacks.
***
Suggestions for improvement / questions (related to decision):
* It should not be a challenge to find more image classification datasets to include in the experimental comparison: SVHN, Fashion MNIST, Imagenet, ... Using these, the paper can either follow the standard train/test methodology *across datasets*, i.e., split the meta-dataset into train/test, and/or provide a more compelling body of evidence for the proposed method. Also, the performance dependence on batchsize amongst the proposed algorithm and competitors should be investigated experimentally.
* The Baydin et al. (2018) algorithm should be added to the set of competitors since it would provide a relatively easy* reference point wrt/ "hypergradient" approaches. I don't agree with the statement in the related work section that this entails "additional computation of gradients." *In the sense that the rule should be straightforward to implement.
* The convergence analysis assumption that the optimal oracle SGD follows typical learning rate regimes motivated by loss plateauing seems to be in direct contradiction to the sentiment expressed in the cited Hoffer et al. (2018) paper that such "rules of thumb" may be misguided. Can the authors discuss the appropriateness of their assumption wrt/ this point? Also, in the convergence analysis, the phrase "in expectation" is used twice. This has a specific probabilistic meaning, but appears to be used heuristically in this section. Can the authors clarify whether this usage is informal or formal? If the latter is true, it would be better to provide a more formal convergence argument that explicitly takes the inherent randomness into account.
***
Editorial comments (not related to decision):
* Introduction: The first two sentences of the second paragraph, particularly the second, would do well to have an accompanying reference or references.
* Proposed method: Even as an informal statement, the second sentence of the second paragraph under the Phase 2 sub-heading is problematic. The proposed method does not "resist" lowering the learning rate "for as long as possible" so much as it doesn't lower the learning rate for a fixed number of epochs (algorithm parameter).
* "Adversarial training" section (5.4): The paper assumes the reader is familiar with the terms "FGSM", "white box", and parameters \epsilon and \alpha since these are referred to w/o description. Perhaps a short (2-3 sentence) description of the adversarial scenario could be added?
* Experiments: It would be good for the paper to include RMSProp and Adagrad results to the experimental tables as these rules are both readily available for use and widely used.
* Experiments: Is the reporting of the peak accuracy standard in the literature?
* Experiments: I want to give the paper credit for performance on CIFAR100, but this is difficult without explicit points of comparison. This can be easily remedied by including SOTA performance values (along with appropriate references) in the tables or text.
* (Potential) Typos:
	Proposed method algorithm description:
		Requirements has a weight decay parameter which seems strange given that the algorithm is performing automated learning rate adaptation...
		The epoch counter is incremented in line 5, but not reset prior to Phase 2. Does this mean that Phase 1 training epochs are counted toward the total (T)?
		Line 7 should be \eta_t <- \eta_0 / 2.
		The patience counter in line 15 is not utilized below.
		Line 23 could/should be an else statement.


Momentum is a simple and popular technique in deep learning for gradient-based optimizers. We propose a decaying momentum (DEMON) rule, motivated by decaying the total contribution of a gradient to all future updates. Applying DEMON to Adam leads to significantly improved training, notably competitive to momentum SGD with learning rate decay, even in settings in which adaptive methods are typically non-competitive. Similarly, applying DEMON to momentum SGD rivals momentum SGD with learning rate decay, and in some cases leads to improved performance. DEMON is trivial to implement and incurs limited extra computational overhead, compared to the vanilla counterparts.

This paper proposed a new decaying momentum rule to further improve neural network training. The idea is motivated by decaying the total contribution of a gradient to all future updates. The authors also extend this idea on to Adam and show that it improved upon the vanilla Adam.
- The intuition of using momentum decaying scheme is not quite clear to me. Why having a linear momentum decay schedule is better? This question is not answered in the paper. I hope the authors could provide more illustrative explanations regarding this.
- This paper focused essentially on empirical evaluations. I do appreciate that the authors conduction various experiments using different dataset and architecture but I do not understand why the authors want to separate the experiments into parts: adaptive methods and adaptive momentum methods. Not to mention the confusing names, it makes no sense to say the same words again for different optimizers. I would suggest the authors to combine the results together for better comparison.
- CM formulation mentioned in the paper is actually different from the commonly refer SGD with momentum implementation, which is the optimizer that is widely used in the community. Therefore, it is important to at least add SGD with momentum as one baseline in the experiments.
- According to https://github.com/kuangliu/pytorch-cifar, Resnet18 on CIFAR10 can achieve at least 93% accuracy using simple SGD with momentum. It seems that the current reported results are not fully optimized. I would suggest the authors to check the parameter settings and make sure all the hyperparameters for baseline methods are fully tuned.
- Aside from SGD with momentum, the experiments part also lacks several important baselines. The author should also consider comparing the AdamW, Padam mentioned in the paper.


Deep convolutional neural networks are known to specialize in distilling compact and robust prior from a large amount of data. We are interested in applying deep networks in the absence of training dataset. In this paper, we introduce deep audio prior (DAP) which leverages the structure of a network and the temporal information in a single audio file. Specifically, we demonstrate that a randomly-initialized neural network can be used with carefully designed audio prior to tackle challenging audio problems such as universal blind source separation, interactive audio editing, audio texture synthesis, and audio co-separation.
To understand the robustness of the deep audio prior, we construct a benchmark dataset Universal-150 for universal sound source separation with a diverse set of sources. We show superior audio results than previous work on both qualitative and quantitative evaluations. We also perform thorough ablation study to validate our design choices. https://iclr-dap.github.io/Deep-Audio-Prior/

This paper presents a method for blind source separation relying on randomly initialized networks to decompose an input audio spectrogram into two components.
The networks are designed to promote temporal contiguity of spectral energy in the estimated signals, which are modulated (in time) by estimated masks.
The proposed method is evaluated on a collection of 150 random mixes of sounds, and performs favorably relative to some standard baseline methods (RPCA, NMF, KAM).
This seems like a promising line of work, but at this point I think the weaknesses of the paper outweigh its strengths (as detailed below).  Some of these points may be addressed during discussion, but I currently lean toward reject.
Strengths of the paper:
- The ideas are interesting, and appear to perform well on a simulated and real(ish) data.
- The authors investigate several variations and applications of source separation, including interactive editing, co-separation, and texture synthesis.
- A small (qualitative) ablation study is included to clarify the importance of different components of the loss function.
Weaknesses of the paper:
- Much of the presentation is vague or opaque.  There is little detail provided about the specific architectural parameters of the model, and the diagram (figure 1) does not appear to match the equations.  Specifically, it's unclear whether S_1^* is a function of S_mixture or not.
- The quantitative evaluation focuses entirely on one (new) dataset with unclear characteristics.  No details are provided about the evaluation protocol, and in particular, the tuning of hyperparameters for the various methods under comparison.  Aggregate statistics are included (mean? SDR, etc), but no notion of variance or error bars are included.  It's not ultimately clear how fair this evaluation is.  There should at minimum be a comparison on a standard source separation or speech enhancement dataset, in addition to the new set presented here.
- Most of the spectrogram figures appear to be upside-down, which is confusing.  The details of the audio processing are omitted: STFT parameters are stated, but not the sampling rate.
- There are numerous typos ("grounth", "spectrogram stokes", etc), indicative of the authors not running a spell-checker on their submission.
Questions for the authors:
- The model itself consists of several competing loss functions, but it seems like they may have a trivial, optimal solution at S1 = S_mix (M_1 = 1) and S_2 = 0 (M_2 = 0 or 1).  As far as I can tell, this solution would trivially minimize each term of equation 9.  (If S_1* does not depend on S_mixture, this may be less of an issue, but the trivial solution may still exist when the driving noise is of sufficiently high dimension and the optimization is run long enough.)  Am I misunderstanding the algorithm, or is there some deeper reason why this solution would not be preferred?


A large fraction of computational science involves simulating the dynamics of particles that interact via pairwise or many-body interactions. These simulations, called Molecular Dynamics (MD), span a vast range of subjects from physics and materials science to biochemistry and drug discovery. Most MD software involves significant use of handwritten derivatives and code reuse across C++, FORTRAN, and CUDA. This is reminiscent of the state of machine learning before automatic differentiation became popular. In this work we bring the substantial advances in software that have taken place in machine learning to MD with JAX, M.D. (JAX MD). JAX MD is an end-to-end differentiable MD package written entirely in Python that can be just-in-time compiled to CPU, GPU, or TPU. JAX MD allows researchers to iterate extremely quickly and lets researchers easily incorporate machine learning models into their workflows. Finally, since all of the simulation code is written in Python, researchers can have unprecedented flexibility in setting up experiments without having to edit any low-level C++ or CUDA code. In addition to making existing workloads easier, JAX MD allows researchers to take derivatives through whole-simulations as well as seamlessly incorporate neural networks into simulations. This paper explores the architecture of JAX MD and its capabilities through several vignettes. Code is available at www.github.com/jaxmd/jax-md along with an interactive Colab notebook that goes through all of the experiments discussed in the paper.

This paper announces a new software package for simulating molecular dynamics which includes close integration with a neural network / machine learning library--the first to do so.  Straightforward access to hardware acceleration (e.g. GPU) is provided for both the simulation and machine learning.
I lean toward accepting this submission.  If it were only about simulation molecular dynamics using hardware accelerators, I would question the appropriateness of the venue, but because it is explicitly intended to support training and usage of learned potential functions, it seems suitable.  Still might be better placed in a physics/chemistry venue, as where most of the references come from and likely where users would, too. The application area is no doubt an important research technique.  The paper is clearly written, with enough specific examples to contrast previous pain points in this line of work against its smoother interface.
All of these points are fine for a package-release/tutorial paper, but for a conference paper, might hope to see these addressed:
Description of the elements of the design of JAX which are useful here are presented, and appear distinct from other AD libraries like Tensorflow or PyTorch, although the authors stop short of explicitly stating which functionality would be more difficult/impossible to support with the possible alternatives (automatic vectorization of the simulations seems like one?).
Limitations of the library and drawbacks of any design decisions (something must be traded at some point?) are not explicitly mentioned.
Despite mentioning numerous existing MD libraries, no performance comparison is drawn against any other.
Could also show some demonstration of running an experiment which has complexity on par with state of the art research?  The bubble raft example is great for illustrative purposes, but it could be better to save some of that for a tutorial and use space to exercise this library on a relevant problem and show performance there.
I took a quick glance at the code on github; it is substantial but not huge, cleanly organized, and is well-documented.


Most of the artificial neural networks are using the benefit of labeled datasets whereas in human brain, the learning is often unsupervised. The feedback or a label for a given input or a sensory stimuli is not often available instantly. After some time when brain gets the feedback, it updates its knowledge. That's how brain learns. Moreover, there is no training or testing phase. Human learns continually. This work proposes a model-agnostic continual learning framework which can be used with neural networks as well as decision trees to incorporate continual learning. Specifically, this work investigates how delayed feedback can be handled. In addition, a way to update the Machine Learning models with unlabeled data is proposed. Promising results are received from the experiments done on neural networks and decision trees.

The paper claims to tackle a semi-supervised continual learning problem where the feedback or the labeled data is delayed and is provided based on the model performance. Authors do not provide standard benchmarks for comparison and no baseline is considered.
The idea of using unlabeled data for continual learning is interesting and to the best of my knowledge this is the first work that suggests using delayed feedback for continual learning but unfortunately they do not consider measuring forgetting and this work seems an online learning method using delayed feedback.
I vote for rejecting this paper due to the following reasons. I have listed the issues in chronological order and not their importance.
1- I start with the writing. The paper, in its current form, needs to be thoroughly proofread and reorganized. The text does not read well and is vague in most parts (for example section 3 and 4). The text is informal in some parts (ex. in Figure 1). There are also grammar errors and typos for which I have found passing my writing through the free version of Grammarly very helpful in getting rid of most such errors.
2- As one of the main motivations for the paper, authors claim humans learn continuously in an unsupervised fashion (paragraph one). I disagree with this statement because we all have been constantly learning from the feedback we have been receiving the environment throughout our lives. For example we all have learned how to walk by falling on the ground multiple times and using the pain signal in our muscles as a negative feedback to correct our movements. Getting corrected while speaking or question answering in our dialogues are examples of receiving feedback from the environment letting our learning behavior receiving lots of supervision.
3- The related work section misses significant number of prior work on continual learning (I have provided a short list at the end [4,5,6] but authors are strongly encouraged to read more on this literature). However, my biggest concern is that I this work should not be introduced as a continual learning algorithm. The proposed method is an online learning method with delayed feedback which has been extensively studied before. Authors should consider citing the pioneering work in this field such as Weinberger & Ordentlich (2002) [2] or Joulani et al from ICML 2013 [3]. Providing comparison to [3] is strongly encouraged. Also note that the citation for “catastrophic forgetting” is wrong and should be corrected to McCloskey & Cohen (1989).
4- The figures and tables do not meet the conventional scientific standards and have to be significantly improved.
5- Authors use softmax probabilities as a confidence score which are known to be uncalibrated by large as deep models are usually overconfident about their predictions. (see [1] for example). Was this investigated at all? Using a calibration technique might be able to help with this [1].
6- On page 4, paragraph 5, the authors claim that “in continual learning instant update is not done”. This is vague to me and I think it is not true because there are plenty of supervised continual learning approaches where the labeled data is available when a task is learned (for example [4,5,6])
7- The experimental setting is not well designed and does not use a standard continual learning setting and there is no baseline included which are very important reasons for rejecting this paper. Authors can benefit from applying their method on standard benchmark datasets commonly used in the literature to provide a fair comparison. Most importantly authors should evaluate their method against prior work.
8- Exploring continual learning for decision trees is completely vague not justified in the paper.
[1] Guo, Chuan, et al. "On calibration of modern neural networks." Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.
[2] Joulani, Pooria, Andras Gyorgy, and Csaba Szepesvári. "Online learning under delayed feedback." International Conference on Machine Learning. 2013.
[3] Weinberger, Marcelo J., and Erik Ordentlich. "On delayed prediction of individual sequences." IEEE Transactions on Information Theory 48.7 (2002): 1959-1976.
[4] Kirkpatrick, James, et al. "Overcoming catastrophic forgetting in neural networks." Proceedings of the national academy of sciences 114.13 (2017): 3521-3526.
[5]Lopez-Paz, David, and Marc'Aurelio Ranzato. "Gradient episodic memory for continual learning." Advances in Neural Information Processing Systems. 2017.
[6] Serrà, J., Surís, D., Miron, M. & Karatzoglou, A.. (2018). Overcoming Catastrophic Forgetting with Hard Attention to the Task. Proceedings of the 35th International Conference on Machine Learning, in PMLR 80:4548-4557


Text generation is a critical and difficult natural language processing task. Maximum likelihood estimate (MLE) based models have been arguably suffered from exposure bias in the inference stage and thus varieties of language generative adversarial networks (GANs) bypassing this problem have emerged. However, recent study has demonstrated that MLE models can constantly outperform GANs models over quality-diversity space under several metrics. In this paper, we propose a quality-diversity controllable language GAN. We illustrate an intimate connection between forward/reverse KL divergences and our principled approach, and demonstrate that our proposed model can achieve quality-diversity control by adjusting a hyper-parameter. Through extensive experiments, we show that the proposed model outperforms MLE models and other GANs variants on synthetic data and is more competitive on real-world tasks such as COCO Image Captions and EMNLP2017 WMT News datasets.

This paper provides a method (loss function) for training GAN model for generation of discrete text token generation. The aim of this loss method to control the trade off between quality vs diversity while generating the text data.
For example,
if original sentence is "The company ’ s shares rose 1 . 5 percent to 1 . 81 percent , the highest since the end of the year ." and the output is "The company ’ s shares rose 1 . 5 percent to 1 . 81 percent , the highest since the end of the year ." then the quality of generation is high and diversity is low.
Pros:
1. The paper is very well written, with importance to the smaller details. It is a very good read for even people who are new to this problem. Especially, I appreciate the part where authors took efforts in writing why a few metrics are not used!
2. The motivation is good and also contributions are explicitly written. The details of the approach are provided clearly.
3. The experiments are provided in two different datasets and also the experiments support the two major claims in the paper.
Cons:
1. The primary concern with this submission is the novelty. The Proposition 1 of using forward-backward JSD based divergence has already been proposed in Li et al. (2019). Also, Li et al. (2019) proposes the entire contribution of this paper. The only difference is the introduction of \pi, which controls the percentage of labelled data to be considered between the generated data and original data. Basically, Li et al. (2019) is a specific case of this paper where \pi = 0.5. Thus, I would consider this paper as one additional experiment in Li et al. (2019) and not a whole paper as such.
2. Also, in the formulation in Eqn 2, the proposition 2 becomes a direct observation when \pi becomes 0 or \pi becomes 1. I would not call this as a proposition but a mere observation of Eqn 2.
3. Thus, taking away proposition 1 (already proposed in Li et al. (2019)) and proposition 2 (which is a mere observation) I do not find any novelty in this paper.
4. From an experiments perspective, Li et al. (2019) performed experiments in 4 datasets: Chinese Poems, MS COCO captions, Obama Speech, and EMNLP2017 WMT news. However, in this paper, results are shown in only two datasets - MS COCO captions and EMNLP2017 WMT news. Was it because that this paper was submitted in haste and/or the results in the other two datasets are not compelling enough to share?
5. The result analysis are poor - the authors have shown only the numbers in the tables, while the interpretation on these numbers and the discussion is left to reviewers discretion. Also, there are no generated examples that the authors are showing in either of the datasets. The authors should further discuss and analyze the results, show generated examples, and explain success and failure cases and the reasons behind them.
Overall, I find the novelty and the experimental analysis of the paper, very weak.


We introduce "TriMap"; a dimensionality reduction technique based on triplet constraints that preserves the global accuracy of the data better than the other commonly used methods such as t-SNE, LargeVis, and UMAP. To quantify the global accuracy, we introduce a score which roughly reflects the relative placement of the clusters rather than the individual points. We empirically show the excellent performance of TriMap on a large variety of datasets in terms of the quality of the embedding as well as the runtime. On our performance benchmarks, TriMap easily scales to millions of points without depleting the memory and clearly outperforms t-SNE, LargeVis, and UMAP in terms of runtime.

The paper proposed ``TriMap’’---a novel dimensionality reduction technique that learns to preserve relative distances among points in a triplet. The paper has done extensive experiments and presents the results in very nice visualizations. The paper is clearly written.
Major merits of this paper are:
1. The proposed method seems effective.
On some datasets (e.g., S-curve), the learned low-dimensional embeddings indeed look very good. And the proposed method has much less runtime than other baselines as shown in table-1.
2. The paper is well written.
However, I am still leaning towards rejecting this submission because the proposed method is lack of necessary justification.
1. Many important technical decisions on this method seem arbitrary, including the parametrization of functions (e.g., s, \omega, \zeta, etc) and values of hyperparameters (e.g., \gamma and \delta). For function forms, the authors should justify why particular parametrization has been chosen; for the hyperparameters, the authors should clearly explain how they are picked---maybe using domain knowledge or tuned on data?
2. The argument for ``global score’’ is not clear enough. There are at least two points that need clarification.
First, ``non-local’’ is not equal to ``global’’. The proposed method indeed considers non-local (or non-near) points while learning embeddings, which helps preserve non-local information. But I am not convinced that the preserved information is actually global. Maybe what helps is to first define ``global’’ in a dimensionality reduction context.
Second, the definition of ``global score’’ depends on another baseline method (i.e. PCA), which seems odd. A principled evaluation (or a score) should be method-independent. What seems right to me is to compute global score (i.e. how much global information has been preserved) by comparing to some statistics in the (high-dim) x space, not to another method.
Moreover, the authors had a strong claim that the proposed ``GS is the only DR performance measure that can reflect this property’’---it doesn’t sound right and why one can’t just use another score which is monotonic wrt the proposed score? The authors mentioned that ``PCA has the lowest possible MRE’’---but this is only right up to the use of a linear transformation and F-norm, so this shouldn’t be a justification for my questions above.
3. What is the reason for the successful runtime?
The authors didn’t clarify why the proposed method is theoretically faster than the baselines.
What I noted is: the authors chose a subset \mathcal{T} for the TriMap method and used \mathcal{T} throughout the paper---is it a typo or is a subset always chosen? If the latter holds, then how was it chosen and how large is it compared to the training data used by other methods? In the end, is the proposed method faster because it uses less data?
Besides the weakness above, I also suggest the authors evaluate their method with some extrinsic evaluations. What’s currently used is only intrinsic---the embeddings are trained to preserve relative distances and are evaluated on a trade-off between local accuracy and the defined global score. It is fine because extensive visualizations are provided and readers can subjectively judge the quality of the learned embeddings. However, the experimental section can be stronger if the authors can show the learned embeddings are better at helping some downstream tasks than other baselines (by preserving non-local information?).


Variational Auto-Encoders (VAEs) are designed to capture compressible information about a dataset. As a consequence the information stored in the latent space is seldom sufficient to reconstruct a particular image. To help understand the type of information stored in the latent space we train a GAN-style decoder constrained to produce images that the VAE encoder will map to the same region of latent space. This allows us to "imagine" the information captured in the latent space. We argue that this is necessary to make a VAE into a truly generative model. We use our GAN to visualise the latent space of a standard VAE and of a β-VAE.

Summary:
This paper proposes a hybrid VAE-GAN model, called the latent space renderer-GAN (LSR-GAN), with the goal to “imagine” the latent space of a VAE, and to improve the decoding and sampling quality of a VAE. First, a VAE-like model is trained, after which the encoder weights are frozen, and the decoder is trained as the generator of a GAN (together with an auxiliary discriminator). The generator loss also contains a reconstruction-like term in the latent space, described by the negative log density of the encoding distribution of the original latent conditioned on the output of the generator: -log q(z|g(z)).
Decision: reject
This paper contains incorrect claims. Leaving those mistakes aside, the experiments don’t lead to new insights and no comparison against other VAE-GAN hybrids is made.
Supporting arguments for decision:
In the introduction the authors state “This is a consequence of the well known evidence lower bound or ELBO objective function consisting of a negative log-probability of generating the original image from the latent representation (this is often implemented as a mean squared error between the image and the reconstruction, although as we argue in Appendix A this term should be proportional to the logarithm of the mean squared error)...”. This statement is surprising and I don’t see how it can be correct. If -log p(x|z) is something like the log of the mean squared error, then the density p(x|z) should be a squared error function, which is not even a valid distribution.
To be more precise, according to the authors, if one takes a Gaussian p(x|z), the reconstruction error should be modeled with
-log p(x|z) = log 1/N sum (x_i - \hat x_i)^2  + const 					(1)
where \hat x_i is a function of z. Note the logarithm on the right-hand side here. In appendix A, the authors observe that most other implementations instead optimize
-log p(x|z) = sum_{i=1}^N (x_i -\hat x_i)^2/(2\sigma^2) + N/2 log(2 pi sigma^2) 		(2)
Where often sigma is set to ½ (so that the last term on the lhs of (2) drops out when gradients are taken). The authors claim the latter is incorrect because sigma should be equal to the (biased) empirical variance. They then insert sigma=empirical variance into (2), take derivatives while ignoring the dependence of sigma on \hat x_i, and then arrive at something like eq. 1. They also argue that those implementations that use sigma=½ are actually optimizing a beta-VAE because of this “incorrect” prefactor of the reconstruction error.
I’m confident that the above claims are not correct.
As an example, using a Gaussian decoder distribution requires parameterizing the mean and variance of that Gaussian distribution as a function of z. Here, setting sigma = ½ (as is commonly done in other literature) is valid, contrary to the above claim. One is always allowed to just set the variance to a constant and ignore its dependence on z. This just leaves you with a less flexible distribution to model p(x|z).
The authors appear to use (1) as the reconstruction error term in the ELBO during VAE optimization, and then play with different prefactors of the KL term similar to beta-VAEs. The combination of the reconstruction error (1) and the prefactor no longer makes this a VAE, but more like a regularized auto-encoder with an unconventional reconstruction error that should not be interpreted as coming from the negative of a log density. Due to the logarithm in front of the mean squared error, the loss function is less sensitive to large errors in reconstruction. It is surprising that the reconstruction error does not cause overflow as log (0) --> - infinity.
Comments on experiments and related work:
- Experiments only show images of reconstructions of a VAE and LSR-GAN, a mean squared error plot of reconstruction errors and an accuracy plot of a classifier evaluated on the reconstructions produced by both models. In terms of MSE the LSR-GAN actually performs worse than their VAE baseline. The authors also train a classifier on cifar-10 and measure its accuracy using the ground truth labels and the reconstructed images of a VAE and LSR-GAN. Here the LSR-GAN performs marginally better than the VAE, but the overall accuracy is very poor.
- No experimental comparison is made against other VAE-GAN hybrids. Related work on hybrid VAE-GANS is discussed in the appendix, not in the main paper.
Other sections:
- Section 3 contains a very long interpretation of the minimum description length (MDL). It is unclear what the goal of this section is, as it does not lead to any results, nor does it help bring the point across as to why the proposed model is good at imagining the latent space of a VAE.
Additional feedback to improve the paper (not part of decision assessment):
The additional loss term of the generator is very similar to what is used in reweighted wake sleep, although this is not mentioned. It could be worth making a connection here.
In the conclusion the authors state “VAEs are often taken to be a pauper’s GAN’. This is not a very scientific statement and can be perceived as insulting for various reasons. Please rephrase this.


Neural reading comprehension models have recently achieved impressive generalisation results, yet still perform poorly when given adversarially selected input. Most prior work has studied semantically invariant text perturbations which cause a model's prediction to change when it should not. In this work we focus on the complementary problem: excessive prediction undersensitivity where input text is meaningfully changed, and the model's prediction does not change when it should. We formulate a noisy adversarial attack which searches among semantic variations of comprehension questions for which a model still erroneously produces the same answer as the original question -and with an even higher probability. We show that -despite comprising unanswerable questions -SQuAD2.0 and NewsQA models are vulnerable to this attack and commit a substantial fraction of errors on adversarially generated questions. This indicates that current models-even where they can correctly predict the answer-rely on spurious surface patterns and are not necessarily aware of all information provided in a given comprehension question. Developing this further, we experiment with both data augmentation and adversarial training as defence strategies: both are able to substantially decrease a model's vulnerability to undersensitivity attacks on held out evaluation data. Finally, we demonstrate that adversarially robust models generalise better in a biased data setting with a train/evaluation distribution mismatch; they are less prone to overly rely on predictive cues only present in the training set and outperform a conventional model in the biased data setting by up to 11% F 1 .

As an extension of recent developments on adversarial attacks and defenses, this paper proposes a simple but effective technique called undersensitivity on machine comprehension task, where the input question is changed but the prediction does not change when it should be. They use two linguistically informed tricks; PoS and NER, to produce the perturbations. In addition to that, several techniques are developed for reducing the adversarial search spaces (Eq 1 and 3) and controlling the level of undersensitivity (Eq 2).
In general, the paper is very well written and clear to read. The formulation of the problem is very straightforward, too. I enjoyed reading the overall paper, especially the experimental results, which provides lots of insights about the techniques. The proposed techniques are simple but they are well-executed in the experiment with reasonable justification. Please find my detailed comments below.
Method.
I appreciate the simplicity of the proposed models with clear motivations. Also, validation of the approaches is well-executed in the experiment.
I like the idea of linguistically-controlled perturbations using PoS and NER. However, there might be many other ways to control it: for example, parsing a sentence using a constituency parser and replacing each phrase with corresponding synonyms/antonyms using WordNet might be interesting. Or, based on the parse, negating the verb might be another way to try. I would expect more linguistically-informed perturbations like these, and I could find some of them from (Kang et al 2018, Ebrahimi et al., 2018). Also, adding a couple of them in the experiment might be interesting to understand the underlying logic of the perturbations.
One major concern of the proposed approach is the sub-optimality by the pre-trained RC model. The undersensitivity (Eq 2) and adversarial search (Eq 3) are calculated by the probability scores predicted by the pre-trained models. This means that producing the new sample x’ is only based on the correctness of the pre-trained model on new samples generated, which sounds to be unreliable. Moreover, using the samples produced by this sub-optimal model may be very limited to produce samples under the sub-optimal space of questions. I wonder how the authors tackle this issue in the experiment.
Experiment
Adversarial attacks should show how an existing system is fragile to be attacked, but at the same time augmenting or adversarially training with them needs to improve its generalization power of the system against the attacks. However, many of the adversarial attack papers mostly focus on the former but not the latter part. In this work, authors showed a result of adversarial training/augmentation but its generalization power on original task (i.e., HasAns case) was not that powerful. The unbiased data setup is interesting but still did not provide any insights about generalization from the adversaries. It would be more convincing to see how this generalization from adversarial attacks can take benefits from bit different tasks such as open-end reading comprehension as a perspective of data augmentation.
I see no comparison with other attacking/defending methods in Tables 3 and 4. Adding the recent models (Ebrahimi et al., 2018, Wallance et al., 2019) may help understand how the proposed models are more effective than other techniques.


We present the first end-to-end verifier of audio classifiers. Compared to existing methods, our approach enables analysis of both, the entire audio processing stage as well as recurrent neural network architectures (e.g., LSTM). The audio processing is verified using novel convex relaxations tailored to feature extraction operations used in audio (e.g., Fast Fourier Transform) while recurrent architectures are certified via a novel binary relaxation for the recurrent unit update. We show the verifier scales to large networks while computing significantly tighter bounds than existing methods for common audio classification benchmarks: on the challenging Google Speech Commands dataset we certify 95% more inputs than the interval approximation (only prior scalable method), for a perturbation of -90dB.

In this work, the authors study the task of building neural network classifiers for audio tasks which can be certified as being resistant to an adversarial attack. One of the contributions of this work is the development of abstract transformers which can be used for the data processing frontend used in typical audio applications. The work also proposes an abstract transformers for LSTMs which is stated to be much faster to use in practice than previous work.
Overall, this work is interesting and I think it would be a great addition to the conference. The paper is generally well written in the initial sections, and the main ideas are very clearly presented. However, there are a number of missing details, particularly in the final sections which discuss the experimental validation. In its present form, I am rating this work as “weak reject”, but I would increase my scores if the authors can improve the final sections in the revised draft.
Main Comments:
1. While I found section 3 to be useful to get an intuition of the proposed method, I still feel that it could be condensed a bit to add in additional details. For example, the authors don’t describe “back-substitution” in the work, which I believe should be described in the main text.
2. A clarification question: When computing provability, the authors state that “We randomly shuffled the test data and then, for every experiment, inferred labels one by one until the number of correctly classified samples reached 100. We report the number of provably correct samples out of these 100 as our provability.” How sensitive was the provability metric to the choice of these 100 test examples? Was the metric computed by repeatedly sampling 100 test cases, for example?
3. The section on “Provable defense for audio classifiers” was not very clear to me. The authors state that “To train, we combine standard loss with the worst case loss obtained using interval propagation.” I was not clear on what the modified loss is. Could the authors please clarify this in the text, preferably a mathematical formulation? Also, I’m curious why these experiments are only conducted on the FSDD set, but not on the GSC set.
4. Figure 5c. Why does the interval analysis technique perform so much worse on the GSC set relative to the FSDD set?  On a related note, it would also be useful to describe some more details about the model architectures for the two tasks.
5. The section on “Experimental comparison with prior work” similarly left me with a number of questions. The authors mention that “We found that, in practice, optimization approach used by POPQORN produces approximations of slightly smaller volume than our LSTM transformer (although non-comparable).” Could these be quantified and reported in the paper. Also, why are the approximation volumes not comparable between the two systems.
Minor comment: It is true that most works in audio classification and speech recognition use processed frontend features such as MFCCs. However, there is also a significant body of work which operates directly on the time-domain signal. Perhaps it would be better to clarify this in the text?
For example:
Pascual S, Bonafonte A, Serra J. SEGAN: Speech enhancement generative adversarial network. arXiv preprint arXiv:1703.09452. 2017 Mar 28.
Sainath TN, Weiss RJ, Senior A, Wilson KW, Vinyals O. Learning the speech front-end with raw waveform CLDNNs. In Sixteenth Annual Conference of the International Speech Communication Association 2015.


Formulating the reinforcement learning (RL) problem in the framework of probabilistic inference not only offers a new perspective about RL, but also yields practical algorithms that are more robust and easier to train. While this connection between RL and probabilistic inference has been extensively studied in the single-agent setting, it has not yet been fully understood in the multi-agent setup. In this paper, we pose the problem of multi-agent reinforcement learning as the problem of performing inference in a particular graphical model. We model the environment, as seen by each of the agents, using separate but related Markov decision processes. We derive a practical off-policy maximum-entropy actor-critic algorithm that we call Multi-agent Soft Actor-Critic (MA-SAC) for performing approximate inference in the proposed model using variational inference. MA-SAC can be employed in both cooperative and competitive settings. Through experiments, we demonstrate that MA-SAC outperforms a strong baseline on several multi-agent scenarios. While MA-SAC is one resultant multi-agent RL algorithm that can be derived from the proposed probabilistic framework, our work provides a unified view of maximum-entropy algorithms in the multi-agent setting.

21st November Update: Thank you for your detailed response. I agree with the suggested future work and revisions to the paper. However, as no updates were made to the submitted paper, I will not be raising my score as the revisions represent significant changes that I cannot support acceptance of without further peer review. I encourage the authors to carefully consider all reviewers advice in updates to the paper for a future submission elsewhere, which I think will significantly improve the paper and its potential impact on the community.
--
This paper contributes a probabilistic framework for multi-agent RL and demonstrates a derivation of multi-agent SAC using it. The framework could be more broadly applicable if it included partial observability (as is often a requirement of multi-agent systems). The derivation could be improved by showing the full working for equation 5, as this would make the work self contained instead of assuming prior knowledge of ELBO by the reader.
The derived algorithm (previously published by Iqbal and Sha [ICML 2019] as noted in the paper) is then evaluated on 4 existing benchmark tasks against the baseline algorithm originally proposed with the environments - MADDPG. The environments represent a good range of multi-agent scenarios of suitable complexity to test modern deep RL algorithms. However, the empirical evaluation and methodology have issues that reduce the significance of their contribution.
On Page 8, it is noted that "the value of alpha_i is empirically tuned for each environment" but for which algorithm was it optimized? It is then noted that the values for alpha were "found using grid search" but details of the range of the search are not included nor details of how any other hyperparameters were set. Were other parameters tuned? If so please report all values searched for both algorithms and how they were set.
At the end of page 8, the caption of Figure 2 concludes "it can be seen that MA-SAC controlled agents outperform MADDPG controlled agents on majority of tasks." This statement is not supported by the graphs in this figure. I suspect Figures 2a and b show no significant difference as the confidence intervals overlap and that Figure 2d is not significantly different throughout training but may be with a small effect size at the current end of training. Figure 2d also looks like training for longer may be beneficial. Therefore, Figure 2c is the only environment that shows a significant improvement. Please provide further evidence that MA-SAC outperforms MADDPG or weaken this conclusion. It would also be interesting to investigate deeper, why MA-SAC shows such higher performance than MADDPG in the Predator-Prey domain.
On Page 9, the conclusion is reiterated and claimed to be in comparison to a state-of-the-art algorithm. However, the benefits of SAC over DDPG have been previously shown both in single agent [Haarnoja et al, ICML 2018] and multi agent domains [Iqbal and Sha, ICML 2019]. A stronger baseline to compare against would improve the significance of any resultant improvements.
The research direction is interesting but the earlier publication of the derived algorithm (Iqbal and Sha, ICML 2019) and the issues discussed above with the experimental results lead me to conclude that the contribution is not yet sufficient to warrant publication. With further work I believe this line of work could lead to a high impact publication, but feel the paper requires more changes than are feasible within the time frame of the ICLR rebuttal period.
Minor Comments:
- Page 4, "the transition function of underlying Markov game" -> the transition function of the underlying Markov game
- Page 9, "in Figure 2c, the red curve corresponds to" -> dark blue curve
- Page 9, "MA-SAC performs at least at par with MADDPG" -> at least on par with
- Page 9, "outperforms it on majority of the tasks" -> outperforms it on the majority of tasks


Convolution is a central operation in Convolutional Neural Networks (CNNs), which applies a kernel to overlapping regions shifted across the image. However, because of the immense amount of correlations in real-world image data, convolutional kernels are in effect re-learning redundant data. In this work, we show that this redundancy has made neural network training challenging, and propose network deconvolution, a procedure which optimally removes pixel-wise and channel-wise correlations before the data is fed into each layer. Network deconvolution can be efficiently calculated at a fraction of the computation cost of a convolution layer. We also show that the deconvolution filters in the first layer of the network resemble the center-surround structure found in biological neurons in the visual regions of the brain. Filtering with such kernels results in a sparse representation, a desired property that has been missing in the training of neural networks. Learning from the sparse representation promotes faster convergence and superior results without the use of batch normalization. We apply our network deconvolution operation to 10 modern neural network models by replacing batch normailization within each. Our extensive experiments show the network deconvolution operation is able to deliver performance improvement in all cases on CIFAR-10, CIFAR-100, MNIST, Fashion-MNIST and ImageNet datasets.

This paper addresses the correlation present in the input data, which may affect learning kernels with redundancy. Therefore, they introduce  a deconvolution mechanism on the input features to remove spatial and channel-wise correlation,  before these features are fed to network layers. They also show the deconvolution effect in the first layer resembles the center-surround effects found in biological neuron and induce sparsity in representation. They draw analogy to batch normalization, and show superiority in terms of  convergence and accuracy.
Overall, the concept of the paper is pretty simple and straightforward - basically it removes the correlation present in the input data, specifically in the case of convolution.The experimental results are promising and show fast convergence over using batch normalization, slightly better accuracy.
However, it seems that the deconvolution to avoid correlation is helping for classification tasks, but may not be applicable for other related tasks such as semantic segmentation, where simply using BN may work.
What is G in the overall complexity? No. of groups? Please define.

One of the questions that comes to mind is considering PCA normalization for the same purpose. Essentially, the step of computing the covariance is the same and decorrelating the data by  using the orthonormal basis vectors, pretty much with a similar motivation. Also, PCA is a linear transformation, which points to a way of learning in the network training. So, my question is what will be the problem of doing PCA transformation and then performing convolutions on that transformed decorrelated data? Maybe layerwise PCA transformation may also help in performance and reduce complexity? I think this discussion should be added to the normalization and whitening section of related works.
Minor note: The recommended page limit is 8 pages, the paper has now 10 pages. For me, Figure 1 is not conveying much information, and can be easily replaced with text. Also, I do not see the need of an extra section for the neurological basis for deconvolution; the brief discussion in the intro is enough.


One very important hyperparameter for training deep neural networks is the learning rate of the optimizer. The choice of learning rate schedule determines the computational cost of getting close to a minima, how close you actually get to the minima, and most importantly the kind of local minima (wide/narrow) attained. The kind of minima attained has a significant impact on the generalization accuracy of the network. Current systems employ hand tuned learning rate schedules, which are painstakingly tuned for each network and dataset. Given that the state space of schedules is huge, finding a satisfactory learning rate schedule can be very time consuming. In this paper, we present AutoLR, a method for auto-tuning the learning rate as training proceeds. Our method works with any optimizer, and we demonstrate results on SGD, Momentum, and Adam optimizers. We extensively evaluate AutoLR on multiple datasets, models, and across multiple optimizers. We compare favorably against state of the art learning rate schedules for the given dataset and models, including for ImageNet on Resnet-50, Cifar-10 on Resnet-18, and SQuAD fine-tuning on BERT. For example, AutoLR achieves an EM score of 81.2 on SQuAD v1.1 with BERT BASE compared to 80.8 reported in (Devlin et al. (2018)) by just auto-tuning the learning rate schedule. To the best of our knowledge, this is the first automatic learning rate tuning scheme to achieve state of the art generalization accuracy on these datasets with the given models.

The authors propose an automatic learning rate schedule based on an explore (always increase LR initially) and then exploit (more typical patience based decay) strategy. The strategy seems to factor in recent understanding of deep learning optimization, and I was very much convinced by the overall idea, even if not fully convinced by the motivation. The main issues with the paper I have is that (1) I does not compare to a strong enough baseline (one cycle), and (2) the schedule does not feel automatic enough to call it "automatic"; see detailed comments. Finally, I found the remarks and analysis regarding width of the minima, and the stipulation that there are more narrow minimas, not substantiated and even contradicatory to some of the results in the literature. Based on this, I am currently leaning towards rejecting the paper. I am willing to raise my score if issues with the experimental setting are addressed.
Detailed comments:
1. I am not convinced that the baseline is strong enough.
While I appreciate the extensive range of architecture and datasets, I am not convinced that the baseline schedule tested against in each setting is actually "state of the art" as stated in the abstract. To the best of my knowledge, the state-of-the-art learning rate schedule, which uses a similar computational budget as the proposed method, is one cycle. In particular, one cycle uses LR range test to select the appropriate starting learning rate and then warmups to it gradually; see [1] for more details (see also [2]).
Note that one cycle includes warming up the learning rate, which is another motivation for including it. As seen for instance in Fig. 4 the proposed AutoLR can increase learning rate initially, while the hand tuned schedules don't. This does not seem to be a fair comparison, given that the state of the art schedules in vision models very often do warm-up the learning rate.
2. The learning rate schedule requires feeding in "seed" learning rate. Is it an automatic learning rate schedule then? I am a bit confused by this claim. Could you please clarify what is the claim about the method?
3. To sum up point 1 and 2, while I like what the paper set out to do, I think that it is key to either (1) demonstrate that the automatic schedule is substantially better than the state of the art schedule, or (2) demonstrate that initial seed learning rate is not an important hyperparameter. If both are not true, then how would you convince the reader to use the method in practice?
4. It is implicitely assumed in the analysis that wide minima are good for generalization. It seems to me that recent evidence points towards the direction that width of the minima is a epiphenomenon, see [3,4,5,6] experiments. I think at least a discussion of (some) of these papers is very important. Also it is worth noting that [5] studied how a high learning rate (or a small batch size) enters wide regions of the loss surface early.
5. "In that respect, an interesting intuitive observation is that a large learning rate can escape narrow minima “valleys” easily (as the optimizer can jump out of them with large steps), however once it reaches a wide minima “valley”, it is likely to get stuck in it (if the “width” of the wide valley is large compared to the step size).". I think [5,8] should be cited here. Both papers studied how learning rate selects minima shape.
6. At the same time, I am not convinced by the experimental data for "We hypothesize that for deep neural networks, narrow minimas far outnumber the wide minimas.":
6a) Taking on the face value that the experiment shows that a low learnring rate can find a wide minima, this doesn't prove the hypothesis. At best (i.e. assuming the indeed there is a strong correlation between width of the low lr minima, and generalization) it proves that low learning rate can find wide minima. It doesn't establish any bound/estimation on the relative number of narrow to wide minima. What if low learning rate by default is driven to narrow minima, and high learning rate is driven to wide minima, but the two are equal in number? I think a similar experimental data would be observed in such a null world.
6b) More importantly, it is not demonstrated that the one experiment that worked better had lower curvature. It seems implicitely assumed that if it worked better then it for sure ended up in a wider minima. I do not think this is a valid reasoning given the weak data for a causal relationship between curvature and generalization (see 2). Could you please report curvature explicitely, rather than making this assumption?
References:
[1] Fast.ai documentation on one cycle method, https://docs.fast.ai/callbacks.one_cycle.html
[2] Arpit et al, Walk with SGD, https://arxiv.org/abs/1802.08770
[3] Golatkar et al, Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence, https://arxiv.org/abs/1905.13277
[4] Yoshida et al, Spectral Norm Regularization for Improving the Generalizability of Deep Learning, https://arxiv.org/pdf/1705.10941.pdf
[5] Jastrzebski et al, On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length, https://arxiv.org/abs/1807.05031
[6] Guiroy et al, Towards Understanding Generalization in Gradient-Based Meta-Learning, https://arxiv.org/abs/1907.07287
[7] Wang et al, Identifying generalization properties in neural networks, https://arxiv.org/pdf/1809.07402.pdf
[8] Wu et al, How SGD Selects the Global Minima in Over-parameterized Learning: A Dynamical Stability Perspective, https://papers.nips.cc/paper/8049-how-sgd-selects-the-global-minima-in-over-parameterized-learning-a-dynamical-stability-perspective.pdf


Classification images and spike triggered analysis have been widely used in psychophysics and neurophysiology to understand underlying mechanisms of sensory systems in humans and monkeys. In this paper, we leverage these techniques to investigate the inherent biases of deep neural networks and to obtain a first-order approximation of their functionality. We emphasize on convolutional neural networks (CNNs) since they are currently the state of the art methods in computer vision and are a good model of human visual processing. In addition, we also study multi-layer perceptrons, logistic regression and recurrent neural networks. Experimenting over three classic datasets, MNIST, Fashion-MNIST and CIFAR-10, we show that the computed bias maps resemble the target classes and when used for classification lead to an over two-fold performance than the chance. Further, we show that classification images can be used to attack a black box classifier and to detect adversarial patch attacks. Finally, we utilize spike triggered averaging to derive filters of CNNs and explore how the behavior of a network changes when neurons in different layers are modulated. Our effort illustrates a successful example of borrowing from neuroscience to study artificial neural networks and highlights the importance of cross-fertilization and synergy across machine learning, deep learning, and computational neuroscience.

This work inspects the bias existing in the neural network classifiers through two techniques from computational neuroscience, the classification image techniques and spike-triggered covariance (STA) analysis. The classification image is generated as the difference map between all the averaged images with predicted classes under noise perturbation. STA is used to generate the eigenvector as the input and response, as a visualization. Both of the tools are the standard psychophysics tools.
The authors use the two tools to visualize the bias learned by various classifiers, CNN, MLP, and logistic regression, trained on three datasets MNIST, Fashion-MNIST and CIFAR 10. There are interesting patterns that emerged from the classifiers trained on MNIST and Fashion-MNIST by the classification image technique, but the result from the CIFAR 10 is not explainable. The classification image is further used to analyze the adversarial attack and defense while STA is used to visualize the internal units.
Strengths:
The paper is well-written and easy to follow. With detailed and complete psychophysics experiments, several interesting properties of the filters and the biases of the neural network classifiers are revealed. It is great to see that more tools from psychophysics can be applied to understand neural network classifiers.
Weakness:
1. One concern I have is that the result on CIFAR-10 is not interpretable. See figure 3. On the other hand, the visualization of filters in higher layers (conv4 and conv6) from the net trained on CIFAR-10 in Figure 8 still looks like edge detectors. There are not so many discussions and explanations on what possibly happened here.
2. The inferior result on CIFAR-10 might indicate that the techniques cannot be generalized to interpret large-scale networks such as AlexNet or resnet, trained on ImageNet or Places. It is 2019 now, I expect to see more experiments on at least AlexNet or VGG, rather than tiny sets MNIST or CIFAR. I will suggest the authors run experiments on AlexNet trained on ImageNet/Places, then we can compare the visualization with the other filter visualization to see the difference.
3. There is no comparison for other bias visualization methods. Following the idea of mean images, I think you can also show the mean images conditional on the predicted labels (average all the testing images with certain predicted labels). How the result from the psychophysics is different from that? How well the proposed methods are able to identify the bias? What other bias cannot be identified? Some discussions on the failure cases would be appreciated


We introduce the notion of lattice representation learning, in which the representation for some object of interest (e.g. a sentence or an image) is a lattice point in an Euclidean space. Our main contribution is a result for replacing an objective function which employs lattice quantization with an objective function in which quantization is absent, thus allowing optimization techniques based on gradient descent to apply; we call the resulting algorithms dithered stochastic gradient descent algorithms as they are designed explicitly to allow for an optimization procedure where only local information is employed. We also argue that a technique commonly used in Variational Auto-Encoders (Gaussian priors and Gaussian approximate posteriors) is tightly connected with the idea of lattice representations, as the quantization error in good high dimensional lattices can be modeled as a Gaussian distribution. We provide experimental evidence of the potential of using lattice representations by implementing a VAE that employs a rectangular lattice and is trained using our results, and contrasting its performance with that of a Gaussian VAE.

This paper proposes a framework for gradient descent optimization of latent variable models where the latent code lies on a (n-dimensional) lattice. After a brief overview of relevant results in lattice theory, a method ("Dithered gradient descent") is proposed to differentiate through lattice quantization by means of an additive "dither" noise variable and a lower bound on the total reconstruction+representation loss using a prior distribution. After elaborating on the relationship between their framework and the VAE literature, the authors go on to give an explanation of the link between the covering property of a lattice and the reconstruction loss. Finally, empirical evidence is given for the better compression and reconstruction properties of a sentence autoencoder with lattice latent codes.
Overall, while the idea presented in this paper is interesting (using lattices as discrete latent variables), I don't think that the paper in its current state is up to standards for ICLR.
First, there is little motivation for using lattices over other discrete latent variable models in the first place (eg. straight-through estimator with Gumbel softmax or VQ-VAE).
Second, while I appreciate the efforts that the authors have gone through to make lattice theory accessible to a profane audience, I think that the presentation (particularly in 2.2) could benefit from significant re-ordering as well as more emphasis on the parallel with the VAE literature. A lot of the concepts introduced there have equivalents in VAE lingo (prior, ELBO) and would benefit from being identified as such as they are introduced (and not just in a following sub-section). I also think that presenting Theorem 1 first is confusing, because we are shown the solution before seeing the problem. I think that it would be much more natural to 1. recall the original objective 2. introduce (and motivate) tithering and the $\hat X$ and then 3. introduce theorem 3 to explain how to get rid of quantization. In terms of global structure the paper is also rather imbalanced with 3 large sections including a monolithic 3 page long introduction. I would advise splitting up as appropriate to give the reader some space to breathe.
Third, and perhaps most importantly, the experimental section is underwhelming. The description of the method used is too sparse and ridden with inaccuracies and vague descriptions. A reader would be hard-pressed to try to reproduce these results from the paper alone. The task itself is rather arbitrary (sentence autoencoding with GRUs), and from what I understand, there is no direct comparison with the relevant literature (eg. Gumbel straight-through or VQ-VAEs). I think that the authors should have at least illustrated the appeal of lattice representation learning on a toy example. Furthermore as stated by the author in the conclusion, the experiments don't even involve actual finite dimension lattices, making most of the paper seem irrelevant in hindsight. Overall this section feels rushed and it is the biggest point against the paper for me.
Finally, the paper is littered with small typos or unfortunate notation collisions which, if individually benign, make parsing this already dense paper harder than needed. Some specific examples are given in the notes at the end of my review.
Notes (in no specific order):
- Second to last paragraph on page 3: "one may want to have $\Delta>0$ be very small [...] but that leads to a larger reconstruction cost". I assume you meant "representation cost".
- $- \log p$ instead of $\log \frac 1 {p}$ to save space
- Voronoi cells are introduced but never referenced.
- Subscripted lowercase f is overloaded, use lowercase p or q for densities. This makes some of the formulas in 2.2 particularly annoying to parse.
- In 2.4 $G_1$ refers to something different than $G_m$ with $m=1$
- The term "dither" is not defined explicitly. The closest I could find is a reference to "the dither value U.". Given the prominence of the term in the paper (and in the name of your proposed method) I think this warrants more attention.
- In the experimental section you justify the use of Gaussian dithering (instead of the uniform dithering used in all the theorems) by "The justification for using Gaussian dithers is partly contained in the high resolution analysis done in Subsection 2.4.". However there is no explicit reference to this fact in 2.4 (in fact the words "gaussian" or "normal" don't even appear in 2.4). The link should be made more explicit somewhere.
- Some sentences seemingly don't make sense: "The representation cost for Gaussian dithering using techniques from Variational Autoencoders; in essence we assume an isotropic unit variance Gaussian as a prior over the representation space and then estimate the KL divergence between the dithered representation conditional on the encoder output and the Gaussian prior." in the experimental section.
- Page 3 codelength: references (eg. the shannon paper). In general a lot of the "It is well-known" statements in the paper should be accompanied by a reference.
- Some references don't have a date (Van den Oord in page 9 for instance)
- the result figure is very hard to read especially on paper. A bit of color as well as larger font/linewidth would go a long way.


In order to make Visual Question Answering (VQA) explainable, previous studies not only visualize the attended region of a VQA model but also generate textual explanations for its answers. However, when the model's answer is 'no,' existing methods have difficulty in revealing detailed arguments that lead to that answer. In addition, previous methods are insufficient to provide logical bases, when the question requires common sense to answer. In this paper, we propose a novel textual explanation method to overcome the aforementioned limitations. First, we extract keywords that are essential to infer an answer from a question. Second, for a pre-trained explanation generator, we utilize a novel Variable-Constrained Beam Search (VCBS) algorithm to generate phrases that best describes the relationship between keywords in images. Then, we complete an explanation by feeding the phrase to the generator. Furthermore, if the answer to the question is "yes" or "no," we apply Natural Langauge Inference (NLI) to identify whether contents of the question can be inferred from the explanation using common sense. Our user study, conducted in Amazon Mechanical Turk (MTurk), shows that our proposed method generates more reliable explanations compared to the previous methods. Moreover, by modifying the VQA model's answer through the output of the NLI model, we show that VQA performance increases by 1.1% from the original model.

Interesting results and analysis but lack of novelty and details.
This paper proposed a novel text explanation method which extracts keywords that are essential to infer an answer from question. The authors proposed VCBS based on CBS and use Natural language inference to identify the entailments when the answer is yes or no. Experiment results show better mean opinion score 100 random samples results compared with the previous method and better VQA binary classifications when flipping based on the NLI results.
Different from [Park et. al. 2018], who collected the explanations, this paper directly use the coco captions dataset as the source for the explanations. One of my major concern about this paper is it actually generate relevant captions with respect to the question and answer instead of real explanations. It's true that correlated caption can sometimes serve as the explanation when they cover the same concept coincidently but, we can not use these captions to explain the reasoning process of VQA.
The technique novelty of the proposed paper is also limited, the major novelty is the VCBS, which seems very similar to CBS. The only difference is VCBS adds relaxed parameters, which seems no technique novelty. Most annotations in Algorithms 1 is also not explained, making the readers hard to follow the actual content. The proposed model, although tied the vqa words with the explanation words, it suffers the same problems as PJ-X model, which didn't consider the VQA attention at all.
The experiment is also weak, considering the results is conduct on 100 samples, there might be significant variance. It's interesting the compared approach is learned based on a different dataset, which makes the results harder to compare. The NLI model results are interesting but for a more fair comparison, I would expect the proposed method compare with a model trained with VQA and coco caption dataset, such as VQA-E.


Generative latent-variable models are emerging as promising tools in robotics and reinforcement learning. Yet, even though tasks in these domains typically involve distinct objects, most state-of-the-art generative models do not explicitly capture the compositional nature of visual scenes. Two recent exceptions, MONet and IODINE, decompose scenes into objects in an unsupervised fashion. Their underlying generative processes, however, do not account for component interactions. Hence, neither of them allows for principled sampling of novel scenes. Here we present GENESIS, the first object-centric generative model of rendered 3D scenes capable of both decomposing and generating scenes by capturing relationships between scene components. GENESIS parameterises a spatial GMM over images which is decoded from a set of object-centric latent variables that are either inferred sequentially in an amortised fashion or sampled from an autoregressive prior. We train GENESIS on several publicly available datasets and evaluate its performance on scene generation, decomposition, and semi-supervised learning.

The paper proposes a generative model for images. There's a probability mask per-pixel per-component (which yields mixing probabilities), and then a set of latents per-component that yield an image. The system is tested on a set of scenes like the GQN dataset, stacks of blocks, and the multi-dsprites dataset. The system is better than MONet, although there are a few lingering questions.
Summary of positives:
+ The factoring of the image into various components is eminently sensible and more work should build in the notion of objects
+ The method is well explained, and I found it easy to understand the entire process.
+ The method does appear to perform better than MONet, and qualitatively produces good results.
Summary of negatives:
- There are a few overclaims that should be fixed: namely that the system is a "generative model of 3D visual scenes" when in reality it is a generative model of images that has a latent space that is perhaps well-positioned to match images generated by 3D scenes composed of objects.
- The method section sets up a few questions that are never answered in the paper: GENESIS--S gets introduced as a baseline to test a hypothesis, but this never gets quantitatively evaluated (indeed GENESIS-S beats GENESIS in 2/3 of the categories where they are compared head-to-head); similarly, the fact that GENESIS produces probabilities is repeatedly sold as an advantage over other methods, but it's never used.
- The experiments are weak: the aforementioned questions aren't tackled, as well as a few other issues.
Overall, I lean ever so slightly towards rejection. The results are good looking, and the method seems well-explained. However, in my view, the manuscript seems to fall short of the mark. I am not, however, strongly opposed to the paper's acceptance. If it is accepted, I would urge authors to address the issues to make their paper have maximum impact.
Big picture things:
-I'm baffled by the repeated claim that this is a generative model of 3D scenes. In the introduction "first object-centric generative model of 3D visual scenes capable of both decomposing and generating scenes". I'm worried I'm just missing something profoundly obvious and critical -- since it seems obvious to me that it's not 3D.
As far as I can tell, x here is a HxWxC image (where C is the number of channels and probably 3); the mixing probabilities are over HxW images, so it's a 2D segmentation system. It may be applied to photos rendered with a perspective camera -- but if the requirement for the system is being applied to images that come via projection, then is faster RCNN a 3D object detector? Are normalized cuts, SLIC superpixels, or any of the other myriad unsupervised segmentation methods from vision then 3D scene segmentation approaches?
-Similarly, it's not really a model for objects -- as can be seen in the multi-colored wall example in Fig 3, and Fig 8, 9. its notion of object is a region with uniform color (presumably due to the Gaussian p(x|z) ). So while it's great that it can model regions of uniform color, the system in large part seems to work (in its current form, reading the current manuscript) because the objects are all one color, and doesn't necessarily decompose the scene into objects.
Unless I'm totally missing something, this sort of claim that should be corrected very quickly. I realize that my personal bias is towards more engineering and harder data and I appreciate that there should be different operating points along the spectrum of how much information is provided. However, I think that it's worth making claims and assumptions clear, rather than claiming to solve the general problem on a specific case because of peculiarities about the specific case.
Method:
-The writing of the method is quite straightforward and written well. The system is clear to me. I largely have no complaints about the method. I do, however, have concerns about the experiments that are done to validate claims in the method section.
-GENESIS-S appears as this baseline "To investigate whether separate latents for masks and component appearances are necessary for decomposition", but then basically disappears. There are, as far as I can see, no qualitative results from it, and it basically appears once quantitatively, in S 4.3 / Table 1, where it appears to be as good if not better than GENESIS. In the appendix it's claimed that GENESIS trains faster than GENESIS-S, but if this is the selling point of GENESIS over GENESIS-S, it would be nice to have at least a little quantification of this.
-There is considerable fuss made over the fact that the system produces probabilities and this is repeatedly mentioned as an advantage over existing systems, but it is never demonstrated that these probabilities are good or useful. This leaves the reader hanging a bit.
Experiments:
Overall the experiments are a little on the weak side.
+The qualitative results are good, but the primary selling point it seems is that the system is better than MONET. This does seem clear, and GENESIS does appear quite a bit better.
- While the results on factoring scenes do appear to be good, there's no quantitative evaluation of this (although the abstract promises it). Section 4.2 essentially says that ARI is bad, without demonstrating in a compelling way that it is. The paper says that this behavior can be seen in Appendix D, but this only shows the oversegmentation and not that this dramatically throws off performance -- I'd expect a figure showing a reversal of expectations -- a clearly worse result getting a clearly better ARI metric.
Additionally, computer vision has been evaluating unsupervised segmentation for close to two decades (BSDS came out in 2001). The authors should look at the metrics in e.g., "Contour Detection andHierarchical Image Segmentation" Arbelaez et al. PAMI 2010. I realize that coming up with metrics is hard, but I think the burden is on the authors to find the metrics to show their conclusions quantitatively.
- The tower stability/height/view experiments seem incomplete. I find them quite interesting, but I'm not entirely sure of what to draw from it. The paper obliquely comments that the Groth paper gets better results with a more complex backbone network and blames it on lack of augmentation, use of a subset of the data, and a reduced resolution. But why not just do straightforward like train a MLP on an even further spatially downsampled image? Random here isn't a particularly compelling baseline or reference point.
Small stuff that doesn't affect my review:
1) It might be worth explicitly pointing out that p_\theta(x_k,z_k^c) is just a Gaussian soon after Eqn 3. It's written after equation 5, but without making the p_\theta explicit. This may help some readers.
2) The tower stability is done with a one-hidden-layer (512) MLP. I'm of the personal opinion that readouts on latent variables are more informative if it's a linear transformation (since this can't do much heavy lifting).
3) The abstract promises semi-supervised learning -- is this the tower experiment? This strikes me more as transfer learning.
-----------------------------
Post review update: I have read the authors' responses, and found them thoughtful and to have answered my questions. I'm happy to accept the paper and would encourage the AC to do so.