Recently, neural-network based forward dynamics models have been proposed that attempt to learn the dynamics of physical systems in a deterministic way. While near-term motion can be predicted accurately, long-term predictions suffer from accumulating input and prediction errors which can lead to plausible but different trajectories that diverge from the ground truth. A system that predicts distributions of the future physical states for long time horizons based on its uncertainty is thus a promising solution. In this work, we introduce a novel robust Monte Carlo sampling based graph-convolutional dropout method that allows us to sample multiple plausible trajectories for an initial state given a neural-network based forward dynamics predictor. By introducing a new shape preservation loss and training our dynamics model recurrently, we stabilize long-term predictions. We show that our model's long-term forward dynamics prediction errors on complicated physical interactions of rigid and deformable objects of various shapes are significantly lower than existing strong baselines. Lastly, we demonstrate how generating multiple trajectories with our Monte Carlo dropout method can be used to train model-free reinforcement learning agents faster and to better solutions on simple manipulation tasks.

Overview:
This paper introduces a method for physical dynamics prediction, which is a version of hierarchical relation network (Mrowca ‘18). HRNs work on top of hierarchical particle-based representations of objects and the corresponding physics (e.g. forces between different parts), and are essentially are graph-convolutional neural networks.
Unlike the original work, the proposed method introduces several improvements: 1. an updated loss function, that adds distance constraints between *all* the particles of the object. 2. recurrent training, when the predictions are fed to the inputs. 3. adding dropout.
Writing:
The paper is relatively well-written and easy to follow.
Evaluation:
Authors compare their model on a dynamics prediction task and seem to outperform the original HRN, especially on longer-term sequences. In addition they report results for trajectory sampling (qualitative) and model-free RL, where using their model as a stochastic simulator seems to have positive impact on agent training.
Decision:
Although the proposed improvements upon HRN generally make sense, it is not clear if those are very significant on their own: adding dropout and recurrent training do not seem particularly novel and, since there is no ablation study, it is hard to see what exactly contributes to the reported improvements.
As for the experimental evaluation, it seems like important baselines are missing, and the model seems to be very sensitive to hyperparameters (see questions). Thus, I am currently leaning more towards a rejection, hence the “weak reject” rating.
Various questions / concerns:
* It is not clear why authors do not provide a comparison to DPI-Nets (Li ICLR‘19). This model seems to be outperforming HRN, and from what it looks like is publicly available: https://github.com/YunzhuLi/DPI-Net. I would encourage authors to provide comparison to this baseline, and potentially on similar sets of experiments, or explain why this comparison would not be possible (which seems unlikely).
* Authors seem to acknowledge that the model is sensitive to the hyperparameter choice (dropout rates), however, there is no numerical evaluation that would help readers understand how critical this choice is for the final performance. Judging from very specific settings in different experiments, this could be a serious concern.
* I find it a bit strange that results in Fig. 7-8 are for random seeds. Is it not possible to just plot an average for e.g. 10 runs?
Update:
I would like to thank the authors for a detailed response!
It seems like there is a common concern about the novelty among reviewers: improvements over HRN are quite incremental. Although authors provide a verbal justification for not comparing to another strong baselines, I do not see why would it not be possible to compare methods in the similar settings, even though that baseline might be more limited.
Generally, if the main contribution is actually only the "stochastic" part and not improved performance, then just adding dropout does not seem like a particularly novel approach to me: whether the convolutions are on graphs or on euclidean domains, this does not change the way dropout is done.


The ability to represent and compare machine learning models is crucial in order to quantify subtle model changes, evaluate generative models, and gather insights on neural network architectures. Existing techniques for comparing data distributions focus on global data properties such as mean and covariance; in that sense, they are extrinsic and uni-scale. We develop a first-of-its-kind intrinsic and multi-scale method for characterizing and comparing data manifolds, using a lower-bound of the spectral Gromov-Wasserstein inter-manifold distance, which compares all data moments. In a thorough experimental study, we demonstrate that our method effectively discerns the structure of data manifolds even on unaligned data of different dimensionality, and showcase its efficacy in evaluating the quality of generative models.

1. Define/explain data manifold: in the abstract, ``data moments" suggests the data is treated as random variables; the authors could explain more on how they make the connection between random variables and manifolds
2. introduction: include an example of ``models having different representation space"
3. introduction: elaborate more on ``unaligned data manifolds"
4. introduction: the distinction between ``intrinsic" and ``extrinsic" can be made more explicit; as currently written, the difference between ``extrinsic"/``multi-scale"/``utilizing higher moments" isn't clear
5. the authors mentioned the hypothesis that high-dimensional data lies on a low-dimensional manifold, but in (3.1) they only considered $\mathbb{R}^d$ without justifying this restriction
6. (3.1) ``$u(t, \cdot) \to \delta(\cdot -x)$" change the dot to a variable
7. (3.1) ``$u(t, x) = 0 \forall x \in \partial X$ and for all $t$" change to ``for all $t$ and $x \in \partial X$
8. equation 1, write ``we restrict our exposition to Euclidean spaces $X = \mathbb{R}^d$" and remove ``$\forall x, x' \in \mathbb{R}^d, t \in \mathbb{R}^+$"
9.  (3.1) the phrase ``a compact X including $\mathbb{R}^d$" is confusing
    as $\mathbb{R}^d$ is not compact under Euclidean topology
10.  (3.1) ``for a local domain with Dirichlet condition $D$" change to ``for a local domain $D$ with Dirichlet condition"
11.  (3.1) ``Definition 1" is more of a proposition/theorem.
12.  (3.1) notation: in the body paragraph the authors use $k(x, x', t)$ but in ``definition 1" they used $k(x, y, t)$
13. (3.1) ``$c$ disconnected components" to ``$c$ connected components"
14. (3.1) the authors could elaborate more about how HKT contains all the information in the graph's spectrum. Currently, they only stated the limit equal to the number of connected components.
15. Github code is helpful but it would be better if they can include an algorithm section
16. A figure of the architecture could be included under the ``putting IMD together" section


Despite its potential to improve sample complexity versus model-free approaches, model-based reinforcement learning can fail catastrophically if the model is inaccurate. An algorithm should ideally be able to trust an imperfect model over a reasonably long planning horizon, and only rely on model-free updates when the model errors get infeasibly large. In this paper, we investigate techniques for choosing the planning horizon on a state-dependent basis, where a state's planning horizon is determined by the maximum cumulative model error around that state. We demonstrate that these state-dependent model errors can be learned with Temporal Difference methods, based on a novel approach of temporally decomposing the cumulative model errors. Experimental results show that the proposed method can successfully adapt the planning horizon to account for state-dependent model accuracy, significantly improving the efficiency of policy learning compared to model-based and model-free baselines.

Learning to Combat Compounding-Error in Model-Based Reinforcement Learning
The authors study the problem of choosing the appropriate planning horizon in model-based reinforcement learning. We want to have a balance between the efficiency of planning into the future and while avoiding the compounding error brought by the learnt model.
And the authors propose to learn an error estimation model that is a function of current state, policy and horizon by estimating the compounding error as a new reward using policy evaluation.
Pros -
I think the novelty in this paper is most appealing for me to vote for a clear acceptance. The growth of model-based reinforcement learning is in a big need of a framework to work with the planning horizon.
The proposed method is a very elegant way of estimating the errors and choose the appropriate horizon.
Still, I think there is a lot of room for improvement, as summarized below.
Cons
- One thing that concerns me is that the maximum horizon H looks very small. A max horizon of 3, 5 or even 10 is still much smaller than the value we usually use in planning based reinforcement learning (MPC in your case). A common choice can be up to 30 or 50 for PETS based algorithms [1].
Does that indicate that the proposed framework is not stable or robust when the horizon is big? Or maybe that scale of horizon is not common in Steve and thus not tried in the experimental section?
- Experiments
For the high dimensional control tasks, only Half-Cheetah and Swimmer was tried. It will generalize the conclusion if the results of other benchmarking environments are tested, including complex tasks such as Humanoid, and easier tasks such as Cart-pole.
- Algorithms
The used baselines in the paper are generally not considered to be state-of-the-art. I guess it is sort-of fair comparison since you add the proposed component on top of the baselines. But extending the current state-of-the-art such as TD3 [2] / PETS
I also suggest adding this benchmarking paper into the related work, which empirically studies the compounding error in different state-of-the-art model-free and model-based reinforcement algorithms.
[1] Chua, K., Calandra, R., McAllister, R., & Levine, S. (2018). Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems (pp. 4754-4765).
[2] Fujimoto, Scott, Herke van Hoof, and David Meger. "Addressing function approximation error in actor-critic methods." arXiv preprint arXiv:1802.09477 (2018).
[3] Wang, Tingwu, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel and Jimmy Ba. “Benchmarking Model-Based Reinforcement Learning.” ArXiv abs/1907.02057 (2019): n. pag.


We consider distributed online convex optimization problems, where the distributed system consists of various computing units connected through a time-varying communication graph. In each time step, each computing unit selects a constrained vector, experiences a loss equal to an arbitrary convex function evaluated at this vector, and may communicate to its neighbors in the graph. The objective is to minimize the system-wide loss accumulated over time. We propose a decentralized algorithm with regret and cumulative constraint violation in O(T max{c,1−c} ) and O(T 1−c/2 ), respectively, for any c ∈ (0, 1), where T is the time horizon. When the loss functions are strongly convex, we establish improved regret and constraint violation upper bounds in O(log(T )) and O( T log(T )). These regret scalings match those obtained by state-of-the-art algorithms and fundamental limits in the corresponding centralized online optimization problem (for both convex and strongly convex loss functions). In the case of bandit feedback, the proposed algorithms achieve a regret and constraint violation in O(T max{c,1−c/3} ) and O(T 1−c/2 ) for any c ∈ (0, 1). We numerically illustrate the performance of our algorithms for the particular case of distributed online regularized linear regression problems.

Summary:
The paper considers a distributed variant of online convex optimization problem over multiple players, where, at each trial t, convex loss_l_t.i is revealed to player i and but evaluated by sum of loss functions sum_i=1^n l_t,i. The players can communicate with their neighborhood and share their decisions. Under the problems setting and some assumption on the neighborhood graph structure, the authors prove regret bounds for convex/strongly-convex losses and full-info/bandit settings. Specifically, the paper allows the algorithm to violate domain constraints but the sum of violation has to be sublinear in the number of trials. They also show the violation bounds simultaneously.
Comments:
The key assumption is Assumption 4, that the players share a doubly-stochastic matrix which is used to mix neighbors’ decisions. This assumption allows to mix all players’ decisions in a long run and the derived regret bounds make sense. The theoretical results are non-trivial.
As a summary, I feel the results are beyond standard previous work and has certain values. Note that I have not evaluated correctness of the results, but the results are likely under Assumption 4.


Spherical CNNs are convolutional neural networks that can process signals on the sphere, such as global climate and weather patterns or omnidirectional images. Over the last few years, a number of spherical convolution methods have been proposed, based on generalized spherical FFTs, graph convolutions, and other ideas. However, none of these methods is simultaneously equivariant to 3D rotations, able to detect anisotropic patterns, computationally efficient, agnostic to the type of sample grid used, and able to deal with signals defined on only a part of the sphere. To address these limitations, we introduce the Gauge Equivariant Spherical CNN. Our method is based on the recently proposed theory of Gauge Equivariant CNNs, which is in principle applicable to signals on any manifold, and which can be computed on any set of local charts covering all of the manifold or only part of it. In this paper we show how this method can be implemented efficiently for the sphere, and show that the resulting method is fast, numerically accurate, and achieves good results on the widely used benchmark problems of climate pattern segmentation and omnidirectional semantic segmentation.

The paper proposes SO(3) equivariant layer, derived from the recently introduced Gauge equivariant CNN framework. The novel contributions are in taking the Gauge equivariance CNN and finding efficient ways to perform logarithmic mapping, parallel transport, and convolution by the equivariant kernel when applied to the sphere. An interpolation scheme for an improved approximation to global SO(3) symmetry is also discussed. Experimental results on spherical MNIST, climate pattern segmentation, and omnidirectional semantic segmentation demonstrate the usefulness of the proposed method for prediction on a sphere.
For the most part, the paper is very clearly written despite the challenging and technical nature of the topic. In particular, the first four pages provide a nice overview of related work and a clear explanation of the Gauge equivariance framework of Cohen et al’19. Two sections that can benefit from further clarification are the proposed "regular non-linearities" (where it would be nice to show in equation why we have equivariance), and equation (4), the logarithmic map (where I had a hard time mapping the discussion in words to the equation).
However, the experiments, while satisfactory, are not impressive: one issue is that the results are mostly compared to the results of two relevant papers by Cohen and colleagues. In recent years, there have been other proposals for deep learning on the sphere, and I wonder why experiments do not try to compare with these works?  (see Kondor et al’18, Coors et al’18, and others cited in the paper.) Moreover, although in theory, the proposed framework improves the Icosahedral CNN of Cohen et al’19 (by directly operating on the sphere rather than an Icosahedral approximation), the practical improvements over the Icosahedral CNN seem to be often marginal (with one exception in spherical MNIST). Do you have any explanation for this? Is there any setup where you expect the proposed approach would give a substantial improvement?


Deep recurrent neural networks perform well on sequence data and are the model of choice. However, it is a daunting task to decide the structure of the networks, i.e. the number of layers, especially considering different computational needs of a sequence. We propose a layer flexible recurrent neural network with adaptive computation time, and expand it to a sequence to sequence model. Different from the adaptive computation time model, our model has a dynamic number of transmission states which vary by step and sequence. We evaluate the model on a financial data set and Wikipedia language modeling. Experimental results show the performance improvement of 7% to 12% and indicate the model's ability to dynamically change the number of layers along with the computational steps.

Summary:
The authors propose Layer Flexible Adaptive Computation Time, an RNN-esque sequence model with varying depth at each time step. The idea is that the model can adaptively choose how much computational effort to spend on each example. The authors evaluate the model empirically on a financial dataset and Wikipedia language modeling, and find that it outperforms a vanilla RNN and the original adaptive computation time (ACT) model.
Unfortunately, the presentation of the idea is unclear, the idea itself is not very novel, and the experimental evaluation is lacking. These weaknesses lead me to vote for a weak reject.
I address specific clarity points below.
In regards to the novelty claim, there have been several developments of depth-based (as opposed to time-based) adaptive computation time in the literature, for example:
[1] McGill et al 2017 "Deciding How to Decide: Dynamic Routing in Artificial Neural Networks"
[2] Bolukbasi et al 2017 "Adaptive Neural Networks for Efficient Inference"
[3] Figurnov et al 2017 "Spatially Adaptive Computation Time for Residual Networks" (this paper is cited by the authors)
These papers do not present sequence models, but the ideas in them readily apply to sequence models. Thus, the main novelty in the authors' paper is handling the different number of hidden states at each timestep via their 'attention mechanism'. While this is definitely a contribution, the unclear presentation and lacking experimental evaluation combine to decrease the value of the paper.
Experimentally, the authors evaluate on a financial time-series dataset and Wikipedia language modeling. They compare to Adaptive computation time and a standard RNN. While the experiments demonstrate a modest improvement over ACT and an RNN, they do not compare on larger, more standard datasets such as the WMT datasets, etc... Additionally, they do not compare with other, more powerful models. Both are required to thoroughly demonstrate their model's effectiveness.
To change my mind the authors would have to (in order of importance):
1) Add experiments on WMT or other bigger datasets and compare with current SOTA models.
2) Thoroughly edit their paper for clarity (specific points below).
The authors may want also add explorations of how much computation time can be saved using their model versus others, as this is very common in the ACT-esque literature.
Specific points:
* While the original ACT paper does use 'mean-field' to denote the convex combination of states at a current timestep, that term has a specific technical meaning different from how it is used here. I would suggest using a different word.
* The introduction is too long and repeats itself in several places.
* There are repeated citations in the literature review.
* In figure 1 sometimes nodes denote functions and sometimes nodes denote outputs. Sometimes the nodes are round and sometimes they are rectangular. Sometimes arrows denote inputs to a function, and sometimes they denote multiplication. These inconsistencies make the figure very hard to decipher.
* The text description of your model is confusing. Specifically, distinguishing between the functions of u_t, \hat{u}_t and \overline{u}_t was difficult.
* In your experiments section the plots are very difficult to interpret because they are phrase as 'improvement over x'. The standard presentation is a table of absolute results. Furthermore, bar charts can be misleading because the scale can make improvements seem bigger than they actually are. The figures should stand alone without having to read the text.


Metagenomic studies have increasingly utilized sequencing technologies in order to analyze DNA fragments found in environmental samples. It can provide useful insights for studying the interactions between hosts and microbes (Methé et al., 2012;Qin et al., 2010), infectious disease proliferation (Chiu & Miller, 2019), and novel species discovery (Nayfach et al., 2019). One important step in this analysis is the taxonomic classification of those DNA fragments. Of particular interest is the determination of the distribution of the taxa of microbes in metagenomic samples. Recent attempts using deep learning focus on architectures that classify single DNA reads independently from each other. In this work, we attempt to solve the task of directly predicting the distribution over the taxa of whole metagenomic read sets. We formulate this task as a Multiple Instance Learning (MIL) problem. We extend architectures used in single-read taxonomic classification with two different types of permutation-invariant MIL pooling layers: a) deepsets and b) attentionbased pooling. We illustrate that our architecture can exploit the co-occurrence of species in metagenomic read sets and outperforms the single-read architectures in predicting the distribution over the taxa at higher taxonomic ranks.

~The authors propose the addition of multiple instance learning mechanism to existing deep learning models to predict taxonomic labels from metagenomic sequences.~
I appreciate the focus area and importance of the problem the authors have outlined. However, I do not think the authors have achieved the conclusions they mention on page 2, as well as other issues throughout the work. I also think the inclusion of the multiple instance learning framework is incremental and does not provide sufficient benefit.
“A new method to generate synthetic read sets with realistic co-occurence patterns from collections of reference genomes”. I do not think there was any systematic analysis of the parameterization of their generative framework. I would appreciate empirical comparison of previous synthetic read generation techniques to the current proposed framework. Also, there is no comparison of the generative framework to real data. How are the parameters chosen in section 3.1.1? Finally, how the authors propose to alleviate bias of composition of databases? Rare species that may be present in abundance in metagenomic data may be swamped out by more common species sequenced again and again in databases.
“A thorough empirical assessment of our proposed model, showing superior performance in prediction the distributions of higher level taxa from read sets.” A comparison to existing alignment-based methods is absolutely required for this work. The authors of GeNet compare to state-of-art Kraken and Centrifuge, and when reading Rojas-Carulla et al., these models have still performed worse than Kraken and Centrifuge, and that should be reported in your assessment.
A few minor points:
-More description of your neural network architecture is needed. I’m not sure what a “ResNet-like neural network” actually means, and how something built for images deals with sequences. There are also different ResNets with different numbers of parameters.
-Why isn’t a 1D convolutional neural network used to process the input sequences? This would make the sequences translation invariant, and would have a similar effect to working with kmers, where k = convolution width.
-It may be useful to interpret the attention mechanism to understand which reads are likely influencing the decision of taxonomic assignment.


Neural networks are able to perform tasks that rely on compositional structure even though they lack obvious mechanisms for representing this structure. To analyze the internal representations that enable such success, we propose ROLE, a technique that detects whether these representations implicitly encode symbolic structure. ROLE learns to approximate the representations of a target encoder E by learning a symbolic constituent structure and an embedding of that structure into E's representational vector space. The constituents of the approximating symbol structure are defined by structural positions -roles -that can be filled by symbols. We show that when E is constructed to explicitly embed a particular type of structure (string or tree), ROLE successfully extracts the ground-truth roles defining that structure. We then analyze a GRU seq2seq network trained to perform a more complex compositional task (SCAN), where there is no ground truth role scheme available. For this model, ROLE successfully discovers an interpretable symbolic structure that the model implicitly uses to perform the SCAN task, providing a comprehensive account of the representations that drive the behavior of a frequently-used but hard-to-interpret type of model. We verify the causal importance of the discovered symbolic structure by showing that, when we systematically manipulate hidden embeddings based on this symbolic structure, the model's resulting output is changed in the way predicted by our analysis. Finally, we use ROLE to explore whether popular sentence embedding models are capturing compositional structure and find evidence that they are not; we conclude by suggesting how insights from ROLE can be used to impart new inductive biases to improve the compositional abilities of such models.

This work focuses on the understanding of Deep NNs and of their generalization ability for compositional tasks and especially for partially composition tasks such as one encounters in natural language processing.
The core idea is to design and to learn a neural model (the ROLE network) able to analyse a target neural network (e.g. an encoder in an seq2seq architecture) by identifying the symbolic structures the target network manipulates (the symbols and their roles) in its representations and the compositional rules it has learned on these. The main motivations are the understanding of what is learnt by a such neural network and getting ideas on which architecture to choose for improving generalization on compositional tasks.

The experimental study shows that ROLE ’s results are relevant when dealing with a fully compositional model by design. This is a kind of sanity check, that ROLE may uncover the ground truth from the data, while no prior information is provided on the nature of the roles.
The experiments on the SCAN dataset concern a standard RNN model learned from data. I am not sure of the nature of RNN used here. How many layers, which activation function ? The experimental section also provides insights on what is learned by the ROLE model and on how its (compositional) representations match the ones learned by the RNN.
As far as in understand, beyond the understanding of the learned representations in RNNs, the paper motivates the work with the expectation that the gained knowledge might be useful for designing better neural nets, with improved generalization ability. Yet i don’t  see clearly what could be done on this line.


Deep neural network image classifiers are reported to be susceptible to adversarial evasion attacks, which use carefully crafted images created to mislead a classifier. Recently, various kinds of adversarial attack methods have been proposed, most of which focus on adding small perturbations to input images. Despite the success of existing approaches, the way to generate realistic adversarial images with small perturbations remains a challenging problem. In this paper, we aim to address this problem by proposing a novel adversarial method, which generates adversarial examples by imposing not only perturbations but also spatial distortions on input images, including scaling, rotation, shear, and translation. As humans are less susceptible to small spatial distortions, the proposed approach can produce visually more realistic attacks with smaller perturbations, able to deceive classifiers without affecting human predictions. We learn our method by amortized techniques with neural networks and generate adversarial examples efficiently by a forward pass of the networks. Extensive experiments on attacking different types of nonrobustified classifiers and robust classifiers with defence show that our method has state-of-the-art performance in comparison with advanced attack parallels.

The paper introduces a new approach to generate adversarial examples for deep classifiers. As opposed to the majority of work on adversarial attack models, which generally limit the attacker on pixel-space distortions measured with respect to an Lp norm, the authors here consider a slightly more general attack model that is a combination of an affine transformation and additive L2 perturbation of the input example.
Finding optimal attacks for this model can be non-trivial (standard due to the highly nonlinear coupling between the affine parameters and the additive perturbation), so the authors instead propose training a surrogate neural network that generates the attack affine-transformation and distortion- parameters sequentially. This can, in principle, be done in a traditional supervised training setup; however, to force the adversarial images to look perceptually close to natural looking images, the authors throw a discriminator loss on top, and train the attack generator network adversarially.
The paper is well-written in general, the idea is intuitive, and the experiments are well-described. However, I have a few concerns that lead to me to give a low score (at least in the first round of reviews).
- Novelty.
Leveraging spatial distortions (or other visually meaningful transformations) to generate adversarial attacks is not a new idea, but the authors seem to have been unaware of this very large body of work. See, for example:
** Engstrom et al, "Exploring the landscape of spatial robustness", ICML 2019
** Poovendran et al, "Semantic adversarial examples", CVPR 2018
** Ho et al, "Catastrophic Child's Play", CVPR 2019
** Joshi et al, "Semantic adversarial attacks", ICCV 2019
among many others.
Using GAN-like transformation models to generate attacks is also not a new idea. A few of the above papers use this approach, and the authors refer to a few other such papers as well.
So as such, the conceptual novelty of the contribution seems to be low (beyond the specific choice of combining affine and L2 perturbations).
- Experimental evaluation.
The authors do a commendable job thoroughly laying out the experimental setup. However, a couple of red flags emerge in the experiments. First, why not look at L-infty perturbations (as opposed to L2)? Second, why not test on more challenging datasets (CIFAR, CelebA, etc) as opposed to simple black/white datasets such as MNIST/Fashion-MNIST? One would imagine that the smaller, simpler datasets are easier to optimize for, and therefore the "amortized" attack generator networks are not necessary here.
- Weakness of theoretical part.
I am not sure the theorem is saying anything strong or useful (since the underlying transformer neural network is assumed to possess infinite capacity). I would suggest just removing it.


We propose a new Stein self-repulsive dynamics for obtaining diversified samples from intractable un-normalized distributions. Our idea is to introduce Stein variational gradient as a repulsive force to push the samples of Langevin dynamics away from the past trajectories. This simple idea allows us to significantly decrease the auto-correlation in Langevin dynamics and hence increase the effective sample size. Importantly, as we establish in our theoretical analysis, the asymptotic stationary distribution remains correct even with the addition of the repulsive force, thanks to the special properties of the Stein variational gradient. We perform extensive empirical studies of our new algorithm, showing that our method yields much higher sample efficiency and better uncertainty estimation than vanilla Langevin dynamics.

I have the rebuttal of the authors, the paper improved indeed and some point on role of M is better clarified now although it is still a bit convoluted. The paper would be stronger if the analysis shows any theoretical advantage to the presented method. I think the author put a good effort in addressing some of my concerns and I m raising my score to 6.

####
Summary of the paper:
The paper proposes stein self repulsive dynamics for sampling from an unnormalized distribution. The method starts by using Langevin dynamic for up to time $Mc$ and then uses those pasts samples to guide the trajectories of the langevin sampling to explore new areas of the densities  using the stein witness function between the current particles  and the past samples( similar to Stein Variational gradient descent).
The paper analyses the mixing properties under standard assumptions of the potential of the Boltzmann distribution and the kernel used in the Stein discrepancy, and shows convergence to the boltzman distribution as the number of particles goes to infinity and the step size goes to zero.
Authors validate their methods and show that it indeed explores on a synthetic example new areas wrt to pure langevin dynamics. Applications in sampling from the posterior of bayesian neural networks and contextual bandits compare the performance of the proposed method to langevin dynamic and pure stein descent favorably.
Clarity/presentation :
The paper is well written and the intuition are well presented .
The notation $\hat{delta}_{M} $ is not great in denoting direct measures, please using another symbol.
My main concerns with the paper are the following:
- the definition of $\bar{\delta}_{M}$ averages only $M$ particles choosen for $M$ time stamps. Something is off here for the continuous approximation to work at each time stamp you need $N$ particles and then you have a past horizon $M$. As $M\to \infty$ this does not matter, but I think in your implementation you are considering at each time step $N$ particles and you average on a horizon of size $M$. is this correct? Please clarify?
- The theorem show only asymptotic behavior and don't quantify the intuition behind the paper , that the "coverage of the samples" is higher.
 Can you for instance bound the wasserstein  distance between the pure langevin and Stein Repulsive dynamic , and between SVGD and your method, as it was done in "The promises and pitfalls of Stochastic Gradient Langevin Dynamics". Basically you can find a coupling between trajectories of your methods and the langevin dynamics and bound the wasserstein distance between the two methods. This will be insightful to see if one would mix faster with respect to the other one.
While the appendices of the paper are lengthy I don't think they explain the most selling point of the method, since they are asymptotic and there is a confusion between the time horizon for the past , and the number of particles.  if $M$ is  $\infty$ in this defintion , we are just running langevin dynamic, and not using Stein witness function. I see an important issue in this definition of the time horizon, I think this time horizon should be finite, and that from time steps less we sample N particles , and we let this $N$ go to infinity.


Autoencoder reconstructions are widely used for the task of unsupervised anomaly localization. Indeed, an autoencoder trained on normal data is expected to only be able to reconstruct normal features of the data, allowing the segmentation of anomalous pixels in an image via a simple comparison between the image and its autoencoder reconstruction. In practice however, local defects added to a normal image can deteriorate the whole reconstruction, making this segmentation challenging. To tackle the issue, we propose in this paper a new approach for projecting anomalous data on a autoencoder-learned normal data manifold, by using gradient descent on an energy derived from the autoencoder's loss function. This energy can be augmented with regularization terms that model priors on what constitutes the user-defined optimal projection. By iteratively updating the input of the autoencoder, we bypass the loss of high-frequency information caused by the autoencoder bottleneck. This allows to produce images of higher quality than classic reconstructions. Our method achieves state-of-the-art results on various anomaly localization datasets. It also shows promising results at an inpainting task on the CelebA dataset.

Summary: The paper proposes to use autoencoder for anomaly localization. The approach learns to project anomalous data on an autoencoder-learned manifold by using gradient descent on energy derived from the autoencoder's loss function. The proposed method is evaluated using the anomaly-localization dataset (Bergmann et al. CVPR 2019) and qualitatively for the task of image inpainting task on CelebA dataset.
Pros:
+ surprisingly simple approach that led to significantly better results.
+ applications to image inpainting, and demonstrates better visual results than using simple VAE.
Concern :
- While I agree that authors have shown relative performance compared to various approaches,  I am not able to map the results of Table-1 to that of Table-3 (second column ROC values) in Bergmann et al. CVPR'19. The setup in two works seem similar. Can the authors please comment to help me understand this difference?
- The proposed approach leads to better performance over the baseline models; it is not clear what is a suitable baseline model for the problem of anomaly localization is?
- The results for image inpainting looks promising. The authors may want to add comparison with existing image inpainting approaches for the reader to better appreciate the proposed approach.


Video prediction models combined with planning algorithms have shown promise in enabling robots to learn to perform many vision-based tasks through only selfsupervision, reaching novel goals in cluttered scenes with unseen objects. However, due to the compounding uncertainty in long horizon video prediction and poor scalability of sampling-based planning optimizers, one significant limitation of these approaches is the ability to plan over long horizons to reach distant goals.
To that end, we propose a framework for subgoal generation and planning, hierarchical visual foresight (HVF), which generates subgoal images conditioned on a goal image, and uses them for planning. The subgoal images are directly optimized to decompose the task into easy to plan segments, and as a result, we observe that the method naturally identifies semantically meaningful states as subgoals. Across four simulated vision-based manipulation tasks, we find that our method achieves more than 20% absolute performance improvement over planning without subgoals and model-free RL approaches. Further, our experiments illustrate that our approach extends to real, cluttered visual scenes.

This paper introduces a hierarchical extension to existing work in vision-based model predictive control. Here, a hierarchical model is optimised to find sub-goals that minimise the planning cost (bottleneck states), so as to allow for improved planning to goal states expressed in higher dimensional state spaces. As expected, results show that this hierarchy improves tasks execution success rates.
The paper is clearly written, although many sub-components of the architecture are described in other articles. While I appreciate that the work is incremental, and that it relies on a number of previously modules, this brevity and the architectural complexity means that the work will be challenging to reproduce or adapt to different tasks.
Nevertheless, I believe the idea is interesting, and the paper does achieve its stated goals of allowing for entirely self-supervised learning without motion primitives, demonstrations and rewards, so makes a useful contribution here.  Compositional planning through sub-goals is a good idea, and this paper adds a useful mechanism to identify suitable sub-goals in a latent space and plan through these.
However, I don't believe that this approach is in any way practically feasible at present, and the paper makes a number of hyperbolic claims that should be toned down to give a more balanced perspective (I appreciate that the authors attempted to do so in their limitations section, but this needs to be done throughout the paper).
Specifically:
1.) Please avoid the use of relative improvement percentages (200% performance improvements in the abstract - the 20% absolute performance is strong enough to stand alone).
2.) The paper oversimplifies the importance of exploration and data collection in the proposed approach. The paper states that a uniform random policy in the continuous action space of the agent works well, and offsets the data collection to "any exploration policy", but this is a key requirement if the proposed approach is to be useful. At present the method seems to be relying on typewriting monkeys writing Shakespeare as a precursor to control. An exploration policy using identified sub-goal of interest may be a particularly valuable extension in future work, but for now the paper glosses over this aspect.
3.) The use of the term "long-horizon tasks" and "manipulation". In line with point 2, my personal feeling is that any task or state that can conceivably be explored and accessed by uniformly sampling from a continuous action space does not qualify as a long-horizon task. From the results and videos, it seems that all of the tasks could be solved by following a single trajectory in 3D space, ie. swing left to knock objects off the table, then right to close the door. Terming this "long-horizon" is a bit of a stretch, as is the notion that flopping about a table bashing into objects counts as "manipulation". Motion planning and trajectory following are not long-term horizon manipulation tasks, solving towers of Hanoi is. Moreover, I appreciate that experiments were conducted on real robot datasets, but this seems to be more of an exercise in latent space anthropomorphism than practical evidence of a feasible control policy.
4.) Limitations regarding latent space expressiveness.  As mentioned in the limitations section of the paper, the proposed approach is heavily reliant on a latent space that fully captures the scene and can roll-out future states sensibly. This is an extremely challenging problem, and one which appears to affect the proposed approach, for example, in the accompanying video (3:.40 - 3:59), the desired goal state contains two objects standing on the table and a closed door, but the policy only closes the door, while objects are knocked over. This seems to indicate that the latent space and planning is unable to learn good object embeddings and spatial representations, and that the type of task that can be solved is along the lines of move forward and to the right when the black blob is on the left of the image, or move down to the left, when there are some coloured blobs on the table.
5.) Limitations around goal state representations . Following on from 4, the identification of important aspects in a given image or latent space represents a major challenge to the proposed approach. Given an image of a desired state, how can the proposed approach identify which elements in the scene are required for success, and which are simply distractors? At present, I see no mechanism by which this could ever be learned in a self-play setting or through an image-based goal state. Eg. what if the image indicated that I want the two objects to be in a specific position, and I never cared about the state of the door?
Despite these limitations, the granularity of the solution is fine for a proof of concept work like this, and is itself a commendable achievement. Unfortunately, the paper's choice of language, relying on terms like "long term horizon" and "manipulation" for simple reaching and pushing tasks exaggerates the state of robot learning, and is potentially misleading to those less familiar with the field.
Despite my gripes, the paper definitely meets the ICLR threshold of "accept if you'd share it with a colleague", and I believe it is is a useful approach to sub-goal identification and a nice piece of work, so I recommend acceptance.


Training generative models like Generative Adversarial Networks (GANs) is challenging for noisy data. A novel curriculum learning algorithm pertaining to clustering is proposed to address this issue in this paper. The curriculum construction is based on the centrality of underlying clusters in data points. The data points of high centrality takes the priority of being fed into generative models during training. To make our algorithm scalable to large-scale data, the active set is devised, in the sense that every round of training proceeds only on an active subset containing a small fraction of already trained data and the incremental data of lower centrality. Moreover, the geometric analysis is presented to interpret the necessity of cluster curriculum for generative models. The experiments on the cat and human-face data validate that our algorithm is able to learn the optimal generative models (e.g. ProGAN) with respect to specified quality metrics for noisy data. An interesting finding is that the optimal cluster curriculum is closely related to the critical point of the geometric percolation process formulated in the paper. Besides, the centrality-accuracy curve in our algorithm provides an effective way of visualizing the model state with curriculum training.

The paper states that training generative models is challenging when there are noisy data points in your training set. To address this, the authors propose to training methodology (or curricula)  where "easier" or more "relevant" points are presented first followed to the model followed by the less relevant ones. The relevance is determined used by calculating centrality on a graph constructed out of the data points.
I lean towards rejecting the paper, primarily because 1) The authors have not provided evidence to claim the noisy data points makes training challenging or characterized anything about under how much noise the training breaks down. 2) Experimental evidence is not convincing  3) There are not of imprecise statements in the paper.
The authors claim (without citation) that training generative models in the presence of noisy data is challenging. How much noise are we talking about? If the entire training set is very noisy, maybe we don't have any hope to learn to generate clean samples, but if it's just a little bit of noise, maybe it's fine. I also understand that when the authors use the word "noise" they don't really mean noise,  but changes in view point etc. Some convincing demonstration that such characteristics of dataset adversely affects the training of generative models will be helpful (in addition to one passing sentence in the experiment section about it).
I am not convinced by any of the experiments. More importantly, it seems like the proposed training curricula is in general valid for a lot of generative models. The experimental evidence is really not convincing. Very limited experiments is done on two datasets only using one particular GAN. To convincingly demonstrate that your training methodology is doing something non-trivial, you should show that this works on multiple generative models on multiple datasets and compare your performance (and visualize some generated samples) against just training blindly on the entire dataset. In addition to this, I don't understand the relevance of some of the results in the paper. For ex, what does Corollary 3 signify?
Some other points:
1. First para. "non-trial' -> non-trivial
2. First para: "model collapse" -> mode collapse
3. 3.2 para 2. "base set that guarantees a proper converge of generative models". What do you mean by "proper convergence"?
4. 3.2 para 2. "moderate compared to m". Again, what is moderate?
5. Why do you use ResNet features (5.1) for distance? Why is this a reasonable or the best metric while computing your graph?
6. "We determine the parameter \alpha of the edge weight by enforcing geometric mean of the weights to be 0.8". It seems very arbitrary to me. Can you justify this choice?


Determining the source of uncertainties in the predictions of AI systems are important. It allows the users to act in an informative manner to improve the safety of such systems, applied to the real-world sensitive applications. Predictive uncertainties can originate from the uncertainty in model parameters, data uncertainty or due to distributional mismatch between training and test examples. While recently, significant progress has been made to improve the predictive uncertainty estimation of deep learning models, most of these approaches either conflate the distributional uncertainty with model uncertainty or data uncertainty. In contrast, the Dirichlet Prior Network (DPN) can model distributional uncertainty distinctly by parameterizing a prior Dirichlet over the predictive categorical distributions. However, their complex loss function by explicitly incorporating KL divergence between Dirichlet distributions often makes the error surface ill-suited to optimize for challenging datasets with multiple classes. In this paper, we present an improved DPN framework by proposing a novel loss function using the standard cross-entropy loss along with a regularization term to control the sharpness of the output Dirichlet distributions from the network. Our proposed loss function aims to improve the training efficiency of the DPN framework for challenging classification tasks with a large number of classes. In our experiments using synthetic and real datasets, we demonstrate that our DPN models can distinguish the distributional uncertainty from other uncertainty types. Our proposed approach significantly improves DPN frameworks and outperform the existing OOD detectors on CIFAR-10 and CIFAR-100 dataset while also being able to recognize distributional uncertainty distinctly.

This work studies the predictive uncertainty issue of deep learning models. In particular, this work focuses on the distributional uncertainty which is caused by distributional mismatch between training and test examples. The proposed method is developed based on the existing work called Dirichlet Prior Network (DPN). It aims to address the issue of DPN that its loss function is complicated and makes the optimization difficult. Instead, this paper proposes a new loss function for DPN, which consists of the commonly used cross-entropy loss term and a regularization term. Two loss functions are respectively defined over in-domain training examples and out-of-distribution (OOD) training examples. The final objective function is a weighted combination of the two loss functions. Experimental study is conducted on one synthetic dataset and two image datasets (CIFAR-10 and CIFAR-100) to demonstrate the properties of the proposed method and compare its performance with the relevant ones in the literature. The issue researched in this work is of significance because understanding the predictive uncertainty of a deep learning model has its both theoretical and practical value. The motivation, research issues and the proposed method are overall clearly presented.
The current recommendation is Weak Reject because the experimental study is not convincing or comprehensive enough.
1.	Although the goal of this work is to deal with the inefficiency issue of the objective function of existing DPN with the newly proposed one, this experimental study does not seem to conduct sufficient experiments to demonstrate the advantages (say, in terms of training efficiency & the capability in making the network scalable for more challenging dataset) of the proposed objective function over the existing one;
2.	Table 1 compares the proposed method with ODIN. However, as indicated in this work, ODIN is trained with in-domain examples only. Is this comparison fair? Actually, ODIN's setting seems to be more practical and more challenging than the setting used by the propose methods.
3.	The evaluation criteria shall be better explained at the beginning of the experiment, especially how they can be collectively used to verify that the proposed method can better distinguish distributional uncertainty from other uncertainty types.
4.	In addition, the experimental study can be clearer on the training and test splits. How many samples from CIFAR-10 and CIFAR-100 are used for training and test purpose, respectively? Also, since training examples are from CIFAR-10 and CIFAR-100 and the test examples are also from these two datasets, does this contradict with the motivation of “distributional mismatch between training and test examples” mentioned in the abstract?
5.	The experimental study can have more comparison on challenging datasets with more classes since it is indicated that DPN has difficulty in dealing with a large number of classes.
Minor:
1. Please define the \hat\theta in Eq.(5). Also, is the dirac delta estimation a good enough approximation here?
2. The \lambda_{out} < \lambda_{in} in Eq.(11) needs to be better explained. In particular, are the first terms in Eq.(10) and Eq.(11) comparable in terms of magnitude? Otherwise,  \lambda_{out} < \lambda_{in} may not make sense.
3. The novelty and significance of fine-tuning the proposed model with noisy OOD training images can be better justified.


Strong inductive biases allow children to learn in fast and adaptable ways. Children use the mutual exclusivity (ME) bias to help disambiguate how words map to referents, assuming that if an object has one label then it does not need another. In this paper, we investigate whether or not standard neural architectures have an ME bias, demonstrating that they lack this learning assumption. Moreover, we show that their inductive biases are poorly matched to lifelong learning formulations of classification and translation. We demonstrate that there is a compelling case for designing neural networks that reason by mutual exclusivity, which remains an open challenge.

*** Increased to weak accept after discussion of merits of ME bias was improved in the paper ***
This paper investigates whether neural networks exhibit a ‘mutual exclusivity (ME) bias’, whereby novel inputs tend to be associated with previously unseen outputs, an inductive bias that is cited to be present in children when learning to associate new words and objects. Via three different sets of experiments, the authors conclude that, under standard training procedures, neural networks in fact display an anti-ME bias: when faced with a novel input, they tend to assign less probability mass to unobserved outputs than is justified by the incoming data. The authors go on to argue that explicitly designing neural networks to reason by mutual exclusivity could lead to faster and more flexible learning. While the experiments across several domains convincingly show the presence of an anti-ME bias in neural network training, I recommend that the paper be rejected because it falls short in demonstrating to what extent performance could be improved by correcting or reversing this bias. The authors pose an interesting hypothesis, but it would gain a lot of credibility if they could provide an empirical analysis of an algorithm that uses ME reasoning to improve learning in a realistic setting or if they could at least perform some quantitative analysis of the effects of ME bias on task performance for a toy example.
Comments / questions:
* The authors duly acknowledge that ME bias is not necessarily desirable in all circumstances, namely in tasks that feature many-to-one mappings, citing polysemy and synonymy in machine translation as examples. Without an empirical example in a realistic setting, it is hard to judge whether the benefits of introducing an ME bias for better classification of new inputs belonging to new classes can outweigh the negatives via a potential increase in misclassification of those belonging to old classes.
* For the synthetic dataset one-to-one mapping of one-hot vectors it is clear that an ME bias would incur an advantage for classification on a zero-shot basis, i.e. an increased an accuracy for the first time a new input is observed. Is ME bias considered to be useful here because (i) we care directly about improving this type of zero-shot classification, or is it (ii) because it's implicitly assumed that a better initial prediction will lead to faster learning on these examples?
    * If it is (i) we care about, then it would be useful to quantify the advantage gained either empirically or analytically. It’s not obvious to me how useful the ME bias would be when there is a large number of classes since the probability assigned to the true novel class with a perfect ME bias would be 1 / (# unseen classes). So at the beginning of training, while there are lots of new classes to benefit from the ME bias, the maximum advantage per example is small, and vice versa at the end of training - how do these conflicting effects balance each other out over the course of training and how do the benefits of ME bias scale with the number of classes? Presumably, whether the benefit scales well will depend on the loss function - in any case, this warrants some analysis/discussion especially given that there are no experiments to test it.
    * The authors say that “ME can be generalized from applying to 'novel versus familiar’ stimuli to instead handling ‘rare versus frequent’ stimuli” and they cite the fact that neural networks take longer to learn from rare stimuli, suggesting that ME could help for reason (ii) above. It’s not obvious to me that ME bias will provide a significant advantage in a one-shot or few-shot learning setting - i.e. how useful is having made a better initial guess (which is random amongst the unseen classes) after you gain some specific information about the target class? Perhaps the benefit is small relative to that of the one-shot or few-shot information gained - I think this needs to be quantified empirically.
* It would be interesting to see a more detailed analysis of the predictions of the image classifiers in section 4.2. When a new image is presented, how is the probability mass distributed across previously seen classes versus across unseen ones? From a strategic standpoint, in a scenario where it is difficult to determine whether a new image belongs to a new class or not, could it plausibly be more sensible for the network to make a strong prediction on its best guess out of previously seen classes (that it knows more about) rather than a scattered prediction on the unseen classes? Admittedly this is a difficult question to quantify, but my point is to question whether the anti-ME bias shown in Figure 6 is necessarily suboptimal, given the difficulty of classifying a new image as a new class or not.
Minor comments /questions not affecting review:
* Is the acronym ME pronounced like the word “me” or is it spelled out “M-E”? If the latter, then all cases of “a ME bias” should be corrected to “an ME bias”.
* Section 4.2 line 3: “sample the class [for the] from a power law distribution"


Weight quantization for deep convolutional neural networks (CNNs) has shown promising results in compressing and accelerating CNN-powered applications such as semantic segmentation, gesture recognition, and scene understanding. Prior art has shown that different datasets, tasks, and network architectures admit different iso-accurate precision values, which increase the complexity of efficient quantized neural network implementations from both hardware and software perspectives. In this work, we show that when the number of channels is allowed to vary such that networks of different precision values have the same model size, lower precision values outperform higher precision ones in a Pareto sense (accuracy vs. model size) for networks with standard convolutions. Relying on comprehensive empirical analyses, we find that the optimal precision value of a convolution layer depends on the number of input channels per output filters and provide theoretical insights for it. To this end, we develop a simple algorithm to select the precision values for CNNs that outperforms corresponding 8-bit quantized networks by 0.9% and 2.2% in top-1 accuracy on ImageNet for ResNet50 and MobileNetV2, respectively.

This paper studies the accuracy vs model-size trade-off of quantized CNNs under different channel width multipliers. The authors demonstrated that while all-to-all convolution works well under low bit settings, depthwise conv needs a different sweet spot. The authors then proceed to use the insight to design quantized cnns that have two different schemes for depthwise and normal conv.
Strength
The paper is well written and motivated. By adding network width to the search space,  using the simple heuristics, the authors provide a better results than previously DRL based search method.
Weakness
One of my main concerns is the direct usage of total number of bits as an equivalent measurement between models. While it is useful to measure the storage cost for weights. The choices of bit width will likely affect the computing cost in a non-trivial way, depending on the target hardware platform. It is unclear whether equal model size would mean equal inference latency in practice (most likely they would not be). Providing empirical implementations of these models will shed light into this question.
These weakness makes it a borderline paper.
Question:
How do you handle batchnorm layer, do you use floating points?
How many bits did you use for accumulating the results?
Update after rebuttal:
I have read the authors' response. I would like to keep my current review as it is. Also note that the authors uses floating pt for batchnorm and 32 bit for accumulation, such additional cost might out-weights the benefit of choosing ultra low bits in the low bits regime, making the study less relevant for practice reasons


How can we efficiently compress Convolutional Neural Networks (CNN) while retaining their accuracy on classification tasks? A promising direction is based on depthwise separable convolution which replaces a standard convolution with a depthwise convolution and a pointwise convolution. However, previous works based on depthwise separable convolution are limited since 1) they are mostly heuristic approaches without a precise understanding of their relations to standard convolution, and 2) their accuracies do not match that of the standard convolution. In this paper, we propose FALCON, an accurate and lightweight method for compressing CNN. FALCON is derived by interpreting existing convolution methods based on depthwise separable convolution using EHP, our proposed mathematical formulation to approximate the standard convolution kernel. Such interpretation leads to developing a generalized version rank-k FALCON which further improves the accuracy while sacrificing a bit of compression and computation reduction rates. In addition, we propose FALCON-branch by fitting FALCON into the previous state-of-the-art convolution unit ShuffleUnitV2 which gives even better accuracy. Experiments show that FALCON and FALCON-branch outperform 1) existing methods based on depthwise separable convolution and 2) standard CNN models by up to 8× compression and 8× computation reduction while ensuring similar accuracy. We also demonstrate that rank-k FALCON provides even better accuracy than standard convolution in many cases, while using a smaller number of parameters and floating-point operations. However, existing approaches based on depthwise separable convolution have several crucial limitations. First, they are heuristic methods, and their relation to the standard convolution is not clearly identified. Second, due to their heuristic nature, generalizing the methods is difficult. Third, although they give reasonable compression and computation reduction, their accuracy is not sufficient compared to that of standard-convolution-based models.
In this paper, we propose FALCON, an accurate and lightweight method for compressing CNN. FALCON overcomes the limitations of the previous methods based on the depthwise separable convolution using the following two main ideas. First, we precisely define the relationship between the standard convolution and the depthwise separable convolution using EHP (Extended Hadamard 1 Under review as a conference paper at ICLR 2020 Product), which is our proposed mathematical formulation to correlate the standard convolution kernel with the depthwise convolution kernel and the pointwise convolution kernel. We then design FALCON by fine-tuning and reordering the results of EHP to improve the accuracy of convolution operations. Second, based on the precise definition, we generalize the FALCON to design rank-k FALCON, which further improves accuracy while sacrificing a bit of compression and computation reduction rates. We also propose FALCON-branch by fitting FALCON into the state-of-the-art convolution unit ShuffleUnitV2 which gives even higher accuracy. As a result, FALCON and FAL-CON-branch provide a superior accuracy compared to other methods based on depthwise separable convolution, with similar compression and computation reduction rates, and rank-k FALCON further improves accuracy, outperforming even the original convolution in many cases. Our contributions are summarized as follows:
• Generalization. We analyze and generalize depthwise separable convolution to our proposed EHP (Extended Hadamard Product) operation. This generalization enables a precise understanding of the relationship between depthwise separable convolution and standard convolution. Furthermore, with fine-tuning operations, it leads to our proposed method FALCON.
• Algorithm. We propose FALCON, a CNN compression method based on depthwise separable convolution. FALCON is carefully designed to compress CNN with little accuracy loss. We also propose rank-k FALCON to further improve the accuracy with a little sacrifice in compression and computation reduction rates. FALCON can be easily integrated into other architectures, and we propose FALCON-branch which combines FALCON with a branch architecture for a better performance. We theoretically analyze the compression and computation reduction rates of FALCON and other competitors.
• Experiments. We perform extensive experiments and show that FALCON 1) outperforms other state-of-the-art methods based on depthwise separable convolution for compressing CNN, and 2) provides up to 8× compression and computation reduction compared to the standard convolution while giving similar accuracies. Furthermore, we show that rank-k FALCON provides even better accuracy than the standard convolution in many cases while using a smaller number of parameters and floating-point operations.
The rest of this paper is organized as follows. Section 2 explains preliminaries. Section 3 describes our proposed method FALCON. Section 4 presents experimental results. After discussing related works in Section 5, we conclude in Section 6.

This paper proposed a model compression method: Falcon and rank-k Falcon. Both are used to compress CNN type of models by replacing standard convolution layer with a compact Falcon or rank-k Falcon layer to compress the model. Falcon's main idea is to decompose the traditional convolution kernel K into two smaller tensors, one is depthwise convolution kernel D and pointwise convolution kernel P. And DP will reconstruct the original kernel K. Since D+P's memory is  D*D*M+N*M which is smaller than the original size D*D*M*N, and thus when N is large, the memory saving could be large. The paper is in general in good writing and very easy to read.
Below I have several concerns/suggestions for this paper:
1: Novelty. What is the main difference between this method and all the other tensor decomposition based methods for CNN compression? There are so many tensor decomposition based methods for CNN, and seems Falcon belongs to one of them. The one (maybe) special for Falcon is that it only decomposes along one dimension. Why this method could perform better than other tensor based decomposition methods(some of them are having even smaller memory footprint as they decompose more dimensions) or Falcon could be one special case of it?
2: In Section 3.3 and 3.4, the proposed Falcon and rank-k Falcon seems is fully recovering the original K, see equations above Theorem 2 and above Section 3.5, should it be minimizing the reconstruction error as other tensor decomposition methods? And how to find the solutions P and D from K? how the model is getting trained if using Falcon or rank-k Falcon? Do you have retrain step after the decomposition of K?
3: In the experiment, no any other standard compression techniques such as quantization, low-rank, weight-sharing, sparse, etc are compared.  This makes us curious about the benefit of the proposed methods over other methods.
In summary, I am mostly worried about the novelty of the paper, and wondering how the model is getting trained, and the comparison with other compression techniques.


Learning useful skills without a manually-designed reward function would have many applications, yet is still a challenge for reinforcement learning. In this paper, we propose Mutual Information-based State-Control (MISC), a new selfsupervised Reinforcement Learning approach for learning to control states of interest without any external reward function. We formulate the intrinsic objective as rewarding the skills that maximize the mutual information between the context states and the states of interest. For example, in robotic manipulation tasks, the context states are the robot states and the states of interest are the states of an object. We evaluate our approach for different simulated robotic manipulation tasks from OpenAI Gym and a navigation task in the Gazebo simulator. We show that our method is able to learn to manipulate the object, such as pushing and picking up, purely based on the intrinsic mutual information rewards. Furthermore, the pre-trained policy and mutual information discriminator can be used to accelerate learning to achieve high task rewards. Our results show that the mutual information between the context states and the states of interest can be an effective ingredient for overcoming challenges in robotic manipulation tasks with sparse rewards. A video showing experimental results is available at https://youtu.be/l5KaYJWWu70.

This paper proposes a self-supervised reinforcement learning approach, Mutual Information-based State-Control (MISC), which maximizes the mutual information between the context states (i.e. robot states) and the states of interest (i.e. states of an object to manipulate). For this, they first split the entire state into two mutually exclusive sets of the context states and the states of interest. Then, the neural discriminator is trained to estimate the (lower-bound of) mutual information between the two states. The (mutual-information) intrinsic reward is computed by the trained neural discriminator, which is used for policy pre-training. Experimental results show that MISC helps to improve the performance of DDPG/SAC and the learned discriminator can be transferred to different environments.
Detailed comments and questions:
- In the paper, the states are represented by only object positions (x, y, z). Is this sufficient? (e.g. velocity is unnecessary?)
- For MISC, the additional assumption is required: the agent should know that which parts of the states are its own controllable state and object's state respectively. Is this additional assumption realistic enough and has it been adopted in other previous works? Is there any way to discriminate robot states and object states automatically?
- Can MISC deal with the problems where the number of objects of interest is more than two? In this case, how can we define mutual information?
- In Eq. (4), T(x_1:N, y_1:N) is assumed to be decomposable into the sum of T(x_t, y_t) / N. Can this make the lower bound (Eq. (3)) arbitrarily loose since the class of functions becomes very limited?
- Detailed experimental setups are missing. e.g. network architecture, hyper-parameters (e.g. I_tran^max), and how they were searched.
- Similarly to the problem of sparse reward, if the robot and the object are far apart and it is difficult to reach the object with random exploration, it would also be difficult to train the mutual information discriminator. How was the discriminator trained? How many time steps were used to train MI discriminator?
- It seems that the MI discriminator learns to estimate the 'proximity' between the robot and the object. Compared to using just a very simple dense reward (e.g. negative L2 distance between the robot and the object), what would be the advantage of using MI discriminator? It would be great to show the comparison between using simple dense reward and MI discriminator for each Push, Pick&Place, and Slide task.
- For the MISC+DIAYN, what if we train the agent using MISC and DIAYN at the same time, instead of pre-training MISC first and fine-tuning DIAYN later?
- It is unclear how MISC-p is performed. Please elaborate on how MISC-p works for prioritization.
- Also, for MISC-r experiments, the weights between the intrinsic reward bonus and the extrinsic reward are not specified in the paper.
- It seems that MISC is beneficial when the robot should get closer to the object for the success of the task. Then, how about the opposite situation? What if the task requires that the robot should 'avoid' the object of interest? Does MISC still work? Is it helpful for the improvement of sample efficiency?
- In order to pre-train the discriminator network, additional (s,a,s') experiences are required, thus it seems difficult to say that it is better for exploration than VIME.
- In section 4.3, what happens if we transfer the learned discriminator to Pick&Place from Push that has a gripper fixed to be closed, rather than the opposite direction (i.e. from Pick&Place to Push)? Does the MISC-t still well work? Can the learned MI discriminator be transferred to different tasks even when the state space is different?


We demonstrate how to learn efficient heuristics for automated reasoning algorithms for quantified Boolean formulas through deep reinforcement learning. We focus on a backtracking search algorithm, which can already solve formulas of impressive size -up to hundreds of thousands of variables. The main challenge is to find a representation of these formulas that lends itself to making predictions in a scalable way. For a family of challenging problems in 2QBF we learn a heuristic that solves significantly more formulas compared to the existing handwritten heuristics.

This paper investigates the problem of predicting the truth of quantified boolean formulae using deep reinforcement learning. In this setting, the problem is formulated as a reinforcement learning task, in which the learner is interacting with a solver (CADET), and its goal is to find a sequence of actions (each associated with a choice of a variable and a value) in order to reach a terminal state as fast as possible. The neural architecture includes a GNN encoder for the input formula, a policy neural net for iteratively assessing the quality of literals, and a final softmax layer for choosing the final literal. Experiments, performed on various 2QBF instances, address several questions such as the ability to compete with existing heuristics (VSIDS) in CADET and to generalize predictions on long episodes or different formulae.
Overall, the paper is well-motivated. The introduction is well-written and explains the interest of learning new heuristics for QBF problems. The learning framework is relatively simple and elegant. Unfortunately, the paper suffers from many clarity issues in the problem formulation, the neural-net architecture, and the experiments. So, it is quite difficult to accept the paper in its current state.
Section 2: In this section, some background knowledge about boolean problems and solvers are provided. Although the second paragraph about CNF formulae and CDCL solvers is well-written, the third paragraph about QBF should be clarified. For QBF formulae, the authors write “The algorithmic problem considered for QBF is to determine the truth of a quantified formula (TQBF)”. Well, this is not an algorithmic problem, but a decision problem. Furthermore, what is TQBF? I guess that the authors are talking about 2-QBF, for which the prenex is of the form $\forall \exists$. Here, the decision task should be explained in more detail using, for example, a game tree for explaining the distinct roles of “for all” variables and “there is” variables. As the decision problem for 2-QBF is more complex than the satisfiability problem for CNF, this point should be emphasized (i.e. the decision problem for 2-QBF is $\Pi_2^P$ complete).
Section 3: The problem of predicting the truth of 2-QBF is formulated as an MDP, where the environment is essentially controlling the input instances (of 2-QBF) and the states of the solver, and the learner’s actions are variable-value selections. First of all, I am not entirely convinced that an MDP is the right choice for specifying this problem. Basically, 2-QBF is a two-player game (“for all” vs “there is”), and the goal of the solver is to play “there is” by finding a satisfying assignment for each possible play (i.e. variable assignment) of the “for all” player. So, a natural framework here would be a stochastic game (SG) which generalizes the MDP framework. Actually, in the present MDP framework there are no distinctions between “for all” variables and “there is” variables. It seems that the learner can choose “for all” variables (which is wrong). Furthermore, the MDP is ill-defined. It is said that a policy is a mapping $\pi: S \times A$. This is ill-defined: what is the range of $\pi$? Usually, a (mixed) policy is a mapping $\pi$ from $S$ into the $|A|$-dimensional simplex. The reward function is also ambiguous: it is using a discount factor $\gamma$ but, unless I missed something, this factor is not clarified in the rest of the paper. Finally, what is an “episode”? In the paper it is said “An episode is the result of the interaction of the agent with the environment”. Well, this is quite unclear. Usually, in an episodic-MDP the state space is partitioned into layers, i.e. $X = \bigcup_{i = 0}^L X_i$, where $X_0$ is a singleton set (the initial state), and $X_L$ specifies the terminal states. Transitions are possible only between consecutive layers.  According to this usual framework, an episode is a sequence of actions made by the agent, starting from $X_0$ and moving forward across the consecutive layers until it reaches a state in $X_L$. For the 2-QBF decision problem, each terminal state in $X_L$ would naturally consists in the affectation of each variable in the prenex to a Boolean value. But this is not clear in the paper, because the authors are saying that “We consider an episode to be complete, if the solver reaches a terminal state in the last step”. This would mean that the number of actions per episodes is capped, and hence, the agent can reach a terminal, yet non-final, decision state.
Section 4: The overall architecture (GNN encoder + Policy Network) is relatively standard, but the choice of the constants for dimension parameters $\lambda_V$, $\delta_L$ and $\delta_C$ is a bit disconcerting. Notably, $\lambda_V$ is used to capture the features of variables. Specifically, it is written that “$\lambda_V = 7$ indicates whether the variable is universally or existentially quantified, whether it currently has a value assigned and whether it was selected as a decision variable already on the current search branch.” But what is the difference between the second feature and the third one? If a variable has already been branched, then it has been assigned, and conversely. Furthermore, if you have 3 boolean features $\lambda_V$ should be fixed to 3. So, why choosing 7? For $\delta_L$ and $\delta_C$, it seems that their values correspond to the “best model”. But how this model is chosen? Did the author perform some grid search to find those values?  Finally, some comments about the last layer of the architecture (softmax function) would be welcome. In the end, we get a probability distribution over literals (agent’s available actions) “after masking illegal actions” (as written by the authors). But what is an illegal action? Is it a literal defined on a universally quantified variable? A literal defined on an already assigned existential variable?
Section 5: In the experiments, the authors are examining four different questions, which are all interesting. But the experimental setup and the reported results are quite unclear. In fact, the experimental setup looks wrong, because if the “Reductions” dataset is taken from Jordan & Kaiser (SAT’13), it consists of formulae for which the prenex is of the form $\exists \forall$”. Unless I am wrong, this is the inverse of the 2QBF problem examined in the present paper  - Jordan and Kaiser were examining a $\Sigma_2^P$-complete problem, while you are examining a $\Pi_2^P$-complete problem. So, did you reverse the quantifiers for making experiments? This should be clarified in the paper. Furthermore, for this dataset which originally consists of 4500 instances, the authors say that “We filtered out 2500 formulas that are solved without any heuristic decisions.” What does this mean? Are all those 2500 formulas containing only universal quantifiers? In the remaining 2000 instances, what is the ratio between universally quantified variables and existentially quantified ones? By the way, how can we get 1835 training instances? 4500 - 2500 - 200 = 1800.
In the experimental results, which formulae have been used to compute the cactus plots? Training instances (1800)? Test instances (200)? Both of them? For training instances, the protocol reported in Section 5.2 is relatively clear. But for test instances, what is the protocol? Is CADET using the best policy trained on the 1800 instances for solving the remaining 200 instances? Furthermore, the notion of “decision limit” is quite confusing. According to Section 5.2, it seems that the decision limit is the horizon of each episode, i.e. the number of calls to the solver CADET using the latest policy estimated by the NN architecture. But this should be clarified unambiguously.


Model-free reinforcement learning (RL) algorithms work well in sequential decision-making problems when experience can be collected cheaply and modelbased RL is effective when system dynamics can be modeled accurately. However, both of these assumptions can be violated in real world problems such as robotics, where querying the system can be prohibitively expensive and real-world dynamics can be difficult to model accurately. Although sim-to-real approaches such as domain randomization attempt to mitigate the effects of biased simulation, they can still suffer from optimization challenges such as local minima and handdesigned distributions for randomization, making it difficult to learn an accurate global value function or policy that directly transfers to the real world. In contrast to RL, model predictive control (MPC) algorithms use a simulator to optimize a simple policy class online, constructing a closed-loop controller that can effectively contend with real-world dynamics. MPC performance is usually limited by factors such as model bias and the limited horizon of optimization. In this work, we present a novel theoretical connection between information theoretic MPC and entropy regularized RL and develop a Q-learning algorithm that can leverage biased models. We validate the proposed algorithm on sim-to-sim control tasks to demonstrate the improvements over optimal control and reinforcement learning from scratch. Our approach paves the way for deploying reinforcement learning algorithms on real-robots in a systematic manner.

This paper studies how a known, imperfect dynamics can be used to accelerate policy search. The main contribution of this paper is a model-based control algorithm that uses n-step lookahead planning and estimates the value of the last state with the prediction from a learned, soft Q value. The n-step planning uses the imperfect dynamics model, which can be cheaply queried offline. A second contribution of the paper is an efficient, iterative procedure for optimizing the n-step actions w.r.t. the imperfect model. The proposed algorithm is evaluated on three continuous control tasks. The proposed algorithm outperforms the two baselines on one of the three tasks. Perhaps the most impressive aspect of the paper is that the experiments only require a few minutes of "real-world" interaction to solve the tasks.
I am leaning towards rejecting this paper. My main reservation is that the empirical results are not very compelling. In Figure 1, it seems like the proposed method (MPQ) only beats that MPPI baseline in BallInCupSparse. The paper seems remiss to not include comparisons to any recent MBRL algorithms (e.g., [Chua 2018, Kurutach 2018, Janner 2019]). Moreover, the tasks considered are all quite simple. Finally, for a paper claiming to "pave the way for deploying reinforcement learning algorithms on real-robots," it seems important to show results on real robots, or at least on a very close approximation thereof. Instead, since the experiments are done in simulation and the true model is known exactly, the paper constructs an approximate model that is a noisy version of the true model.
I do think that this paper is tackling an important problem, and am excited to see work in this area. I would consider increasing my review if the paper were revised to include a comparison to a state-of-the-art MBRL method, if it included experiments on more complex task, and if the proposed method were shown to consistently outperform most baselines on most tasks.
Minor comments:
* The derivation of the update rule in Section 4.2 was unclear to me. In Equation 17, why can the RHS not be computed directly? In Equation 21, where did this iterative procedure come from?
* "Reinforcement Learning" -- No need to capitalize
* "sim-to-real approaches … suffer from optimization challenges such as local minima and hand designed distributions." -- Can you provide a citation for the "local minima" claim? The part about "hand designed distributions" seems to have a grammar error.
* "Model Predictive Control" -- No need to capitalize.
* "Computer Go" -- How is this different from the board game Go?
* "Entropy-regularized" -- Missing a period before this.
* Eq 1 -- The "\forall s_0" suggests that this equation depends on s_0. Can you clarify this dependency?
* "stochastic gradient descent" -- I believe that [Williams 1992] is a relevant citation here.
----------------------UPDATE AFTER AUTHOR RESPONSE------------------------
Thanks for the detailed response!  The new experiments seem strong, and help convince me that the method isn't restricted to simple tasks. I'm inclined to agree that it's not overly onerous to assume access to an imperfect model of the world. Because of this, I will increase my vote to "weak accept."
In the next version, I hope that the authors (1) include multiple random seeds for all experiments, and (2) student the degree to which model misspecification degrades performance (i.e., if you use the wrong model, how much worse do you do?).


Training reinforcement learning policies in partially observable domains with sparse reward signal is an important and open problem for the research community. In this paper, we introduce a new sequence-level intrinsic novelty model to tackle the challenge of training reinforcement learning policies in sparse rewarded partially observable domains. First, we propose a new reasoning paradigm to infer the novelty for the partially observable states, which is built upon forward dynamics prediction. Different from conventional approaches that perform self-prediction or one-step forward prediction, our proposed approach engages open-loop multi-step prediction, which enables the difficulty of novelty prediction to flexibly scale and thus results in high-quality novelty scores. Second, we propose a novel dual-LSTM architecture to facilitate the sequence-level reasoning over the partially observable state space. Our proposed architecture efficiently synthesizes information from an observation sequence and an action sequence to derive meaningful latent representations for inferring the novelty for states. To evaluate the efficiency of our proposed approach, we conduct extensive experiments on several challenging 3D navigation tasks from ViZDoom and DeepMind Lab. We also present results on two hard-exploration domains from Atari 2600 series in Appendix to demonstrate our proposed approach could generalize beyond partially observable navigation tasks. Overall, the experiment results reveal that our proposed intrinsic novelty model could outperform several state-of-the-art curiosity baselines with considerable significance in the testified domains.

This paper extends the prediction-error based model by Pathak et. al., 2019 by learning a forward (and inverse) dynamics model for predicting a state feature multiple steps into the future (say, K-steps) given an open loop sequence of K actions as opposed to 1 step into the future, with the caveat that instead of using learnable state features, a random network is used for computing state features, similar to Random Network Distillation (RND) by Burda et. al., 2019. Also, their inverse dynamics models predicts the entire sequence of actions up to K steps. Experiments on VizDoom point-navigation tasks show that the proposed model does better than baselines as rewards get sparser. Ablations are provided to justify the choice of K in multi-step prediction, the choice of inverse dynamics and the choice of RND state features.
My decision is weak reject as:
1. The paper does a good job at clearly explaining their model, presenting results on relevant experiments and baseline comparisons for their model and justifying each modeling choice with ablations.
2. The novelty in their contribution is moderate - the idea of long-term future prediction is not new (e.g.: Ke et. al., 2019) (but using it for giving a curiosity bonus is new), the architecture choice is not significantly new.
3. I expected a more detailed ablation for the choice of K, given that “multi-step” has major emphasis in the paper, but Figure 7(a) only shows ablations for 3 vs 1 step predictions.
Elaborating on (3):
- My major concern is that the gap between 1-step prediction and 3-step prediction (in Figure 7 (a)) is not significant. Note that the version of their model with 1-step predictions does not completely reduce to Pathak et. al. 2019’s ICM model, as RND state features are used. I feel that it may be the case that the 1-step version of the proposed model is actually good enough to beat all the baselines and adding multiple steps gives marginal gains. This hypothesis needs to be verified by the authors with more experiments - I would like to see all the main experiments have an additional baseline of 1-step predictions.
- Only two values of K are tested - 1 and 3, what happens with larger values of K?
Other comments:
The motivation for using long-term predictions to “infer more meaningful novelty” is fine on it’s own but seems to conflict with the choice of random network (RND) state features. Random features imply that a random “hash” of the observations is being computed which has no reason to have similar features for two nearby states. If there is any small amount of noise in the state transitions, this would mean that predicting far into the future is practically impossible given that the random feature of slightly incorrect states will be very different. Can the authors give reasons/motivations as to why such a model would work in the case of stochastic transitions and K is large or will it be brittle to stochasticity?
References:
All references are same as those cited in paper.


Neural networks are widely used in Natural Language Processing, yet despite their empirical successes, their behaviour is brittle: they are both over-sensitive to small input changes, and under-sensitive to deletions of large fractions of input text. This paper aims to tackle under-sensitivity in the context of natural language inference by ensuring that models do not become more confident in their predictions as arbitrary subsets of words from the input text are deleted. We develop a novel technique for formal verification of this specification for models based on the popular decomposable attention mechanism by employing the efficient yet effective interval bound propagation (IBP) approach. Using this method we can efficiently prove, given a model, whether a particular sample is free from the under-sensitivity problem. We compare different training methods to address under-sensitivity, and compare metrics to measure it. In our experiments on the SNLI and MNLI datasets, we observe that IBP training leads to a significantly improved verified accuracy. On the SNLI test set, we can verify 18.4% of samples, a substantial improvement over only 2.8% using standard training.

-- Overall --
This submission tackles to verify the “under-sensitivity” problem of neural network models in the natural language inference by ensuring modes do not become more confident in the predictions when arbitrary subsets of words from the input text are deleted. The authors developed new verification approaches based on decomposable attention mechanism with interval bound propagation (IBP), which can prove the under-sensitivity issue given a model and a particular sample. The experimental results on SNLI and MNLI show that the proposed approach leads to a much improved verified accuracy.
-- In general, “under-sensitivity” is a very critical problem for applying neural models in natural language understanding where powerful neural networks tend to capture spurious correlations from the biased datasets. This submission formulates “under-sensitivity” as a mathematical specification and then try to verify it with IBP verification. Although the used technique IBP is not new, it would interesting to have the verification in NLI models.
-- Section 5 is a bit unclear how to compute the IBP for deleting several words, and what is the output. It would be better to have a clear example for how this was computed.
-- As the author mentioned, the verification of under-sensitivity can also be done by using beam-search, although it is costly and not accurate. IBP is another more efficient option, but not the optimal neigher. Maybe consider to change the title as “efficient verification”?
-- Specific Questions --
The entire paper builds on decomposable attention. Is the same approach also applicable to other model types, or only single layer attention-based models?
Also, how this methods work for other NLI or NLU tasks?
In experiments, how the data augumentation penalize the model with a loss for specification violation? What does the equation look like?
Can you explain a bit more for IBP-training? How that hinge loss applies to the objective function? Is the IBP training differentiable?