Acceleration for reinforcement learning methods is an important and challenging theme. We introduce the Anderson acceleration technique into the value iteration, developing an accelerated value iteration algorithm that we call Anderson Accelerated Value Iteration (A2VI). We further apply our method to the Deep Q-learning algorithm, resulting in the Deep Anderson Accelerated Q-learning (DA2Q) algorithm. Our approach can be viewed as an approximation of the policy evaluation by interpolating on historical data. A2VI is more efficient than the modified policy iteration, which is a classical approximate method for policy evaluation. We give theoretical analysis of our algorithm and conduct experiments on both toy problems and Atari games. Both the theoretical and empirical results show the effectiveness of our algorithm.

This paper introduces the "Anderson mixing" ideas from the broader literature on general fixed-point problems to the specific problem of finding the fixed-point to the Bellman optimality equations for a Markov Decision Processes. The general idea is to summarizes the history of previous iterates (value functions in this case) by finding of convex combination which also minimizes the residuals. The authors provide a solution for when an iterate is no longer representable by a convex combination of the recent history by simply bypassing the interpolation step and replacing it with a usual value iteration step. Using the intuition developed in the MDP case, they then adapt their DP algorithms to the learning case by substituting exact (tabular) value functions with deep function approximators. Experimental results are presented in 3 games from the ALE environment.
The jump from the DP formulation to the learning case is rather abrupt, and lacks sufficient motivation. The way the paper is currently structured is 50-50: 50% of the contribution is the DP view of the proposed method while the remaining half comes from the deep formulation (and experiments). I think that I would have preferred to see the entire paper being dedicated to the DP point of view, followed by a more principled Approximate DP analysis in the simpler linear case. Dedicating the remaining of the paper to the deep formulation almost feels like a missed opportunity to fully developing the theory initiated in the first section. But then of course the price to pay would be a paper which would be less aligned with the "representation learning" aspect of the conference. My main concern is that extending this technique to the deep setting mare involve some serious interference with other mechanisms already at play. It is very difficult to explain if the observed improvement come from the underlying DP basis or as a secondary effect of architectural and algorithmic considerations.
To my knowledge, this is the first attempt at using Anderson mixing in the MDP framework. However, I would appreciate if the authors could survey previous attempts (if any) by other authors, or more generally existing results in the literature on non-linear fixed-point methods.  You may find relevant work by consulting the recent Zhang, O’Donoghue and Boyd paper (2018).
# Detailed comments
> Puterman 2014
The 2014 edition is likely to be a re-print of the 1994 which is commonly cited. I would double-check to see if there is any difference in the content between the 2014 and 1994 edition. If not (and just a re-print) I would cite the 1994 edition which is more widely recognized.
> Citations for VI and PI
You should cite Bellman 1957 and Howard 1961 (not Puterman). For exact references, see bibliographical remarks in Puterman.
> Citation for Modified policy iteration
Please cite original paper(s) by Puterman and Brumelle ~1978. See bibliographical remarks in Puterman 1994 (or 2014) for the origins of MPI.
>  via the Neumann expansion
truncated
> computationally inefficient for complex decision problems
Compared to what? More efficient than full PI for sure
> Page 2, notation for $\Gamma_\pi$ vs $\Gamma$
I suggest using a different notation for the (linear) policy evaluation operator vs the Bellman optimality one. The subscript "_\pi$ is easy to miss.
> converges much faster with K
Define K
> In most cases, we can
In reinforcement learning, we can
> value iteration can be finished
Finished ?
> value iteration can be finished by estimating Γ(v) through sampling.
We are no longer in the realm of DP, but more stochastic approximation methods. This isn't quite VI anymore. I would be more careful when jumping from one setting to the other.
> provided the sampling estimations are accurate enough
The approach described so far does not involve any sampling.
> This modification is based on the observation that the recent successive policies do not
So far, the mixing equations (3) and (4) only describe the evaluation case. You haven't mentioned yet how you plan to combine this into a more general control algorithm where successive (changing) policies are generated.
> the solution can be written explicitly as
Please cite where this comes from (or provide proof inline or appendix)
> while PI is similar to Newton’s method
Cite Puterman and Brumelle for the original work on showing the connection between PI and Newton's method.
> except that the tangent line is replaced with a secant line.
Please explain this intuition: how you obtain this geometric interpretation.
Also, the secant method being an analogue to quasi-Newton methods, and policy iteration being Newton's method, there is an opportunity to better develop and explain those parallels.


Many machine learning systems are implemented as pipelines. A pipeline is essentially a chain/network of information processing units. As information flows in and out and gradients vice versa, ideally, a pipeline can be trained end-to-end via backpropagation provided with the right supervision and loss function. However, this is often impossible in practice, because either the loss function itself may be non-differentiable, or there may exist some non-differentiable units. One popular way to superficially resolve this issue is to separate a pipeline into a set of differentiable sub-pipelines and train them with isolated loss functions. Yet, from a decision-theoretical point of view, this is equivalent to making myopic decisions using ad hoc heuristics along the pipeline while ignoring the real utility, which prevents the pipeline from behaving optimally. In this paper, we show that by converting a pipeline into a stochastic counterpart, it can then be trained endto-end in the presence of non-differentiable parts. Thus, the resulting pipeline is optimal under certain conditions with respect to any criterion attached to it. In experiments, we apply the proposed approach -reinforced pipeline optimization -to Faster R-CNN, a state-of-the-art object detection pipeline, and obtain empirically near-optimal object detectors consistent with its base design in terms of mean average precision.

Pros:
+ Improving joint training of non-differentiable pipelines is a meaningful and relevant problem
+ Using the stochastic computation graph structure to smooth a pipeline in a structured way is a plausible idea
Cons:
+ The main result of the paper concerning sufficient conditions for optimality of the method seems dubious
+ It is not obvious why this method would outperform simple baselines, and baselines for joint training were tried
+ The notation seems unnecessarily bloated and overly formal
+ The exposition spends too much time on prior work, too little on the contribution, and the description of the contribution is confusing
The submission describes a method for smoothing a non-differentiable machine learning pipeline (such as the Faster-RCNN detector), so that gradient-based methods may be applied to jointly train all the parameters of the pipeline.  In particular, the proposal involves recasting the pipeline as a stochastic computation graph (SCG), adding stochastic nodes to this graph, and then using REINFORCE-style policy gradients to perform parameter learning on the SCG.  It is claimed that under certain conditions, the optimal parameters of the resulting SCG are also optimal for the original pipeline.  The method is applied to optimizing the parameters of Faster-RCNN.
I think making non-differentiable pipelines differentiable is an intuitively appealing concept.  A lot of important, practical machine learning systems fall into this category, so devising a nice way to do global parameter optimization for such systems could potentially have significant impact.  In general, we can’t hope to make much meaningful progress on the problem of optimizing general nonlinear, differentiable functions, but it is plausible that a method that targets key non-differentiable components for smoothing—such as this paper—could outperform a generic black-box optimizer.  So, I think the basic idea here is plausible and addresses an important problem.
Unfortunately, I think this work loses sight of that high-level goal: to me, the key question is whether the proposed approach outperforms any other simple method for global parameter optimization in the presence of nonlinearities and nondifferentiability.  The paper fails to answer this question because no baselines for global parameter optimization were tried.  We can just treat the pipeline as a black box mapping parameters to training set performance, and so any black-box optimization method can be applied to this problem.  It is not clear that the proposed method would outperform an arbitrary black box optimization method such as simulated annealing, Nelder-Mead, cross-entropy method, etc.
I think there are also much simpler methods in a similar vein to the proposed method that might also perform just as well as the proposal.  One key conceptual issue here is that reducing the problem to a reinforcement learning problem, as the submission does, is not much of a reduction at all.  First, if the goal is to do global parameter optimization, then we don’t really have to smooth the pipeline itself: we can just smooth the black box mapping parameters to performance, and then optimize that with SGD.  There are many ways to do this--if we want to use policy gradient, we can just express the problem as something in this form:
min_\phi E_{\theta ~ q_\phi} C(\theta)
where C is the black-box mapping parameters \theta to a performance index (such as mean AP), q_\phi is a distribution over parameters (e.g., Gaussian), and \phi are the distribution parameters (e.g., mean, covariance of the Gaussian).  We can then optimize this using REINFORCE policy gradients.
If we want to really smooth the pipeline itself, then it is also easy to do this by devising a suitable MDP and then applying REINFORCE with the usual MDP structure.  We simply identify the state s_t at time t with the output of the t’th pipeline stage, introduce a new ‘action’ variable a_t representing a ’stochastified output’, and trivial dynamics (P(s_{t+1} | s_t, a_t) = \delta(s_{t+1} - a_t)).  If the policy is a Gaussian (P(a_t | s_t) = N(a_t; s_t, \Sigma)), then this is similar to relaxing the constraint that one stage’s output is equal to the input of the next stage, and somehow quadratically penalizing their difference.  In fact, there is a neural network training method based explicitly on this penalization view [A], and it would make yet another great baseline to try.
In fact, the proposed method is essentially similar to what I have just described, but it is unfortunately described in an overcomplicated way that obscures the true nature of the method.  I think the whole SCG framework is overkill here.  Too much of the paper is spent just rehashing the SCG framework, and the very heavy notation again just obscures the essential character of the method.
If there were, as the paper claims, some interesting condition under which the method produces solutions that are optimal under the original pipeline, that would be remarkable and interesting.  However, I have serious doubts about this part of the paper.  The key problem is the statement that “It follows that c(k_c, DEPS_c - k_c) = c(…) + z_c”.  The paper seems to be claiming that if E z = 0, then c(k + z) = c(k) + z, which can’t possibly be true in general.
The heavy and opaque notation makes it very difficult to understand this section.  Perhaps it would help to consider a very simple example.  Suppose we want to minimize E_{x ~ q} c(y(x)) (where x ~ q means x is distributed as q).  We can introduce only one new stochastic node (k = y + z), between y and c.  Clearly c(y + z) is not generally equal to c(y) + z, even if E z = 0.
In summary, I think the submission needs a lot of work on multiple axes before it can make a significant impact.  The most important issues are a complete lack of relevant baselines and the dubious claims about sufficient conditions for optimality.  The idea could have merit, but it needs to be carefully compared and motivated with respect to existing work (such as [A]) as well as the simple baselines I have mentioned.  The presentation also needs to be revised to find the simplest expression of the method and to focus on the interesting parts.
[A] Taylor, Gavin, et al. "Training neural networks without gradients: A scalable admm approach." International Conference on Machine Learning. 2016.


Self-attention has been widely used to model the sequential data and achieved remarkable results in many applications. Although it can be used to model dependencies without regard to positions of sequences, self-attention is seldom applied to non-sequential data. In this work, we propose to learn representations of multi-field categorical data in prediction tasks via self-attention mechanism, where features are orderless but have intrinsic relations over different fields. In most current DNN based models, feature embeddings are simply concatenated for further processing by networks. Instead, by applying self-attention to transform the embeddings, we are able to relate features in different fields and automatically learn representations of their combinations, which are known as the factors of many prevailing linear models. To further improve the effect of feature combination mining, we modify the original self-attention structure by restricting the similarity weight to have at most k non-zero values, which additionally regularizes the model. We experimentally evaluate the effectiveness of our self-attention model on non-sequential data. Across two click through rate prediction benchmark datasets, i.e., Cretio and Avazu, our model with top-k restricted self-attention achieves the state-of-the-art performance. Compared with the vanilla MLP, the gain by adding self-attention is significantly larger than that by modifying the network structures, which most current works focus on.

Quality:
- In 4.4, the authors have vigorously explored the space of hyperparameters. However, they do not describe how to determine the hyperparameters, e.g., set aside a validation set from a part of the training set and determine the hyperparameters using this validation set, while the authors split the two datasets into only training and test sets, respectively. Without this procedure, the results may overfit to the test set via repeated experiments. Even though the used datasets are of few-million, this procedure guarantees a minimum requirement for a reliable outcome from the proposed model. I firmly recommend the authors to update their results using a validation set to determine the hyperparameters and then report on the test set. Please describe these experimental details to ensure that the performed experiments are valid.

Clarity:
- Overall, the writing can be improved via proof-reading and polishing the sentences. In Introduction section, "there is little work applying..." can be specified or rephrased with "it is underexplored to apply", and "input features are not independent" can be specified on what there are not independent. Moreover, the last two sentences in the second paragraph in the Introduction section is unclear what the authors want to argue: "The combinations in linear models are then made by cross product over different fields. Due to the sparsity problem, the combinations rely on much manual work of domain experts."
- The authors use top-k restriction (Shazeer et al., 2017) to consider sparse relationships among the features. For this reason, have you tried to use the L1 loss on the probability distributions, which are the outputs of softmax function?
- In 4.5, the authors said, they "are in most concern of complementarity." What is the reason for this idea and why not the "relevance"?
- In Table 4, I'm afraid that I don't understand the content (three numbers in parenthesis) of the third column. How does each input x_i or x_j, or a tuple of them get their own CTR?
Originality and significance:
- They apply self-attention to learn multiple categorical features to predict Click-Through-Rate (CTR) with a top-k non-zero similarity weight constraint to adapt to their categorical inputs. Due to this, the scientific contribution to the corresponding community is highly limited to providing empirical results on the CTR task.
- The authors argue that "most of current DNN-based models simply concatenate all feature embeddings"; however, this argument might be an over-simplified statement for the existing models in section 2.
- Similar works can be found but missed to cite: [1] proposes a general framework to self-attention to exploit sequential (time-domain) and parallel (feature-domain) non-locality. [2] learns bilinear attention maps to integrate multimodal inputs using skip-connections and multiple layers on top of the idea of low-rank bilinear pooling.
Pros:
- Strong empirical results on two CTR tasks using the previous works of self-attention and top-k restriction techniques.
Cons:
- This work fairly lacks its originality since the proposing method heavily relies on the two previous works, self-attention and top-k restriction. They apply them to multiple categorical features to estimate CTR; however, their application seems to be monotonic without a novel idea of task-specific adaptation.
Minor comments:
- In Figure 1, "the number of head" -> "the number of heads".
[1] Wang, X., Girshick, R., Gupta, A., & He, K. (2018). Non-local Neural Networks. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'18).
[2] Kim, J.-H., Jun, J., & Zhang, B.-T. (2018). Bilinear Attention Networks. In Advances in Neural Information Processing Systems 32 (NIPS'18).


FLUCTUATION-DISSIPATION RELATIONS FOR STOCHASTIC GRADIENT DESCENT
The notion of the stationary equilibrium ensemble has played a central role in statistical mechanics. In machine learning as well, training serves as generalized equilibration that drives the probability distribution of model parameters toward stationarity. Here, we derive stationary fluctuation-dissipation relations that link measurable quantities and hyperparameters in the stochastic gradient descent algorithm. These relations hold exactly for any stationary state and can in particular be used to adaptively set training schedule. We can further use the relations to efficiently extract information pertaining to a loss-function landscape such as the magnitudes of its Hessian and anharmonicity. Our claims are empirically verified.

The paper introduces the concept of fluctuation-dissipation relations to stochastic gradient descent. These relations hold for certain observables in physical systems in equilibrium. In the context of SGD as a non-equilibrium process with a stationary density, they allow to quantify how far away this process is from its stationary state.
One of the strengths of the paper is that it works in the discrete-time formalism and uses the master equation, as opposed to other recent works that used the continuous-time limit of SGD to derive related (yet different) results. Furthermore, the formalism does not even rely on a locally quadratic approximation of the loss function, or on any Gaussian assumptions of the SGD noise. To the best of my knowledge, all of this is very innovative. Ultimately, the authors propose a practical algorithm to adaptively lowering the learning rate based on testing fluctuation-dissipation relations.
This is an interesting paper which I recommend to accept. It not only shows new theoretical results, but also conforms their validity in real-world experiments.
I have only a few questions / comments:
1. In Eq. 17 and others where the scalar product of theta and grad(f) occurs, is it implicitly assumed that the optimum of f is at theta=0?
2. In Fig. 2, the distinction between solid and dotted curves could be made better visible.
3. For completeness, it would be good to add the following citation:
Stephan Mandt, Matthew D. Hoffman, and David M. Blei. "Continuous-time limit of stochastic gradient descent revisited." NIPS 2015 Workshop on Optimization for Machine Learning.


FUNCTIONAL VARIATIONAL BAYESIAN NEURAL NETWORKS
Variational Bayesian neural networks (BNNs) perform variational inference over weights, but it is difficult to specify meaningful priors and approximate posteriors in a high-dimensional weight space. We introduce functional variational Bayesian neural networks (fBNNs), which maximize an Evidence Lower BOund (ELBO) defined directly on stochastic processes, i.e. distributions over functions. We prove that the KL divergence between stochastic processes equals the supremum of marginal KL divergences over all finite sets of inputs. Based on this, we introduce a practical training objective which approximates the functional ELBO using finite measurement sets and the spectral Stein gradient estimator. With fBNNs, we can specify priors entailing rich structures, including Gaussian processes and implicit stochastic processes. Empirically, we find fBNNs extrapolate well using various structured priors, provide reliable uncertainty estimates, and scale to large datasets.

-- Paper Summary --
The primary contribution of this paper is the presentation of a novel ELBO objective for training BNNs which allows for more meaningful priors to be encoded in the model rather than the less informative weight priors featured in the literature. This is achieved by way of introducing a KL measure over stochastic processes which allows for priors to take the form of GP priors and other custom variations. Two approaches are given for training the model, one inspired by GANs, and a more practical sampling-based scheme. The performance of this training scheme is validated on a variety of synthetic and real examples, choosing Bayes by Backprop as the primary competitor. An experiment on contextual bandit exploration, and an illustrative Bayesian optimisation example  provided in the supplementary material showcase the effectiveness of this method in applications where well-calibrated uncertainty is particularly pertinent.
-- Critique --
This paper makes important strides towards giving more meaningful interpretations to priors in BNNs. To the best of my knowledge, the KL divergence between stochastic processes that gives rise to an alternate ELBO has not been featured elsewhere, making this a rather interesting contribution that is supplemented by suitable theorems both in the main text and supplementary material. The introductory commentary regarding issues faced with increasing the model capacity of BNNs is particularly interesting, and the associated motivating example showing how degeneracy is countered by fBNN is clear and effective.
The GAN-inspired optimisation scheme is also well-motivated. Although the authors understandably do not pursue that scheme due to the longer computation time incurred (rendering its use impractical), it would have been interesting to see whether the optimum found using this technique is superior to the sampling based scheme used throughout the remainder of the paper. The experimental evaluation is also very solid, striking an adequate balance between synthetic and real-world examples, while also showcasing fBNNs’ effectiveness in scenarios relying on good uncertainty quantification.
In spite of the paper’s indisputable selling points, I have several issues with some aspects of this submission. For clarity, I shall distinguish my concerns between points that I believe to be particularly important, and others which are less significant:
- Monte Carlo dropout (Gal & Ghahramani, 2016), and its extensions (such as concrete dropout), are widely-regarded as being one of the most effective approaches for interpreting BNNs. Consequently, I would have expected this method to feature as a competitor in your evaluation, yet this method does not even get a cursory mention in the text.

- The commentary on GPs in the related work paints a dour picture of their scalability by mostly listing older papers. However, flexible models such as AutoGP (Krauth et al, 2017) have been shown to obtain very good results on large datasets without imposing restrictions on the choice of kernels.
 - The regression experiments all deal with a one-layer architecture, for which the proposed method is shown to consistently obtain better results. In order to properly assess the effectiveness of the method, I would also be interested in seeing how it compares against BBB for deeper architectures on this problem. Although the authors cite the results in Figure 1 as an indicator that BBB with more layers isn’t particularly effective, it would be nice to also see this illustrated in the cross-dataset comparison presented in Section 5.2.
 - Furthermore, given that all methods are run for a fixed number of iterations, it might be sensible  to additionally report training time along with the results in the table. This should reflect the pre-processing time required to optimise GP hyperparameters when a GP prior is used. Carrying out Cholesky decompositions for 1000x1000 matrices 10k times (as described in Section 5.2.2) does not sound insignificant.
- The observation regarding the potential instability of GP priors without introducing function noise should be moved to the main text; while those who have previously worked with GPs will be familiar with such issues, this paper is directed towards a wider audience and such clarifications would be helpful for those seeking to replicate the paper’s results. On a related note, I would be keen on learning more about other potential issues with the stability of the optimisation procedure, which does not seem to be discussed upfront in the paper but is key for encouraging the widespread use of such methods.
- The paper contains more than just a handful of silly typos and grammatical errors - too many to list here. This single-handedly detracts from the overall quality of the work, and I highly advise the authors to diligently read through the paper in order to identify all such issues.
 - The references are in an absolute shambles, having inconsistent citation styles, arXiv papers cited instead of conference proceedings, etc. While this is obviously straightforward to set right, I’m nonetheless disappointed that this exercise was not carried out prior to the paper’s submission.
 - The theory presented in Appendix A of the supplementary material appears to be somewhat ‘dumped’ there. Given that this content is crucial for establishing the correctness of the proposed method, linking them more clearly to the main text would improve its readability and give it a greater sense of purpose. I found it hard to follow in its current state.
** Minor **
 - In the introduction there should some mention of deep Gaussian processes which are implicitly a direct competitor to BNNs, and can now also be scaled to millions and billions of observations (Cutajar et al. 2017; Salimbeni et al. 2017). The former is particularly relevant to this work since the architecture can be assimilated to a BNN with special structure for emulating certain kernels.
 - Experiment 5.1.1 is interesting, and the results in Figure 2 are convincing. I would also be interested in seeing how fBNN performs when the prior is misspecified however, which may be induced by using a less appropriate GP kernel. This would complement the already provided insight on using tanh vs ReLU activations.
 - The performance improvement for the experiment on large regression datasets is quite subdued, so it might be interesting to see how both methods compare against each other when deeper BNN architectures are considered.

- With regards to Appendix C.2, which order arccosine kernel is being used here? One can easily draw similarities between the first order arccosine kernel and NN layers with ReLUs, so perhaps it would be useful to specify which order is being used in the experiment.

- Given that the data used for experiments in Appendix C.3 effectively has grid structure, I would be interested in seeing how KISS-GP performs on this task. There should be easily accessible implementations in GPyTorch for testing this out. Given how GPs tend to not work very well on image completion tasks due to smoothness in the kernel, this comparison may also be in fBNNs favour.
- Restating the basic architecture of the BNN being used for the contextual bandits experiment in the paper itself would be helpful in order to avoid having to separately check out Riquieme et al (2018) to find such details.
- I wonder if the authors have already thought about the extendability of their proposal to more complex BNN architectures such as Bayesian ConvNets?
-- Recommendation --
Whereas several ICLR submissions tend heavily towards validation by way of empirical evaluation, I find that the theoretic contributions presented in this paper are by themselves interesting and well-developed, which is very commendable. However, there are multiple telling signs of this being a rushed submission, and I am less inclined to argue ardently for such a paper’s acceptance. Although the paper indeed has its strong points, both in terms of novelty and varied experimental evaluation, in view of this overall lack of finesse and other concerns listed above, I think that the paper is in dire need of a thorough clean-up before being published.
Pros/Cons summary:
+   Interesting concepts that extend beyond empirical fixes.
+   Defining more interpretable priors is a very pertinent topic in the study of BNNs.
+   The presented ideas could potentially have notable impact.
+   Illustrative experiments and benchmark tests are convincing.
-   Not enough connection to MC dropout.
-   Choice of experiments and description of stochastic processes overly similar to other recent widely-publicised papers. It feels on trend, but consequently also somewhat reductive.
-   More than a few typos and grammatical errors.
-   Presentation is quite rough around the edges. The references are in a particularly dire state.


The joint optimization of representation learning and clustering in the embedding space has experienced a breakthrough in recent years. In spite of the advance, clustering with representation learning has been limited to flat-level categories, which oftentimes involves cohesive clustering with a focus on instance relations. To overcome the limitations of flat clustering, we introduce hierarchically clustered representation learning (HCRL), which simultaneously optimizes representation learning and hierarchical clustering in the embedding space. Specifically, we place a nonparametric Bayesian prior on embeddings to handle dynamic mixture hierarchies under the variational autoencoder framework, and to adopt the generative process of a hierarchical-versioned Gaussian mixture model. Compared with a few prior works focusing on unifying representation learning and hierarchical clustering, HCRL is the first model to consider a generation of deep embeddings from every component of the hierarchy, not just leaf components. This generation process enables more meaningful separations and mergers of clusters via branches in a hierarchy. In addition to obtaining hierarchically clustered embeddings, we can reconstruct data by the various abstraction levels, infer the intrinsic hierarchical structure, and learn the level-proportion features. We conducted evaluations with image and text domains, and our quantitative analyses showed competent likelihoods and the best accuracies compared with the baselines.

This paper proposes using a variant of the nested CRP as a prior on the latent space of a variational autoencoder. The authors demonstrate that this approach is able to simultaneously learn a meaningful latent representation of high-dimensional data (text and images) and do hierarchical clustering in that space.
Pros:
* The high-level idea is compelling.
* The empirical results are compelling, and the evaluation is thorough.
Cons:
* The prose is pretty rough. The paper is full of sentences like "VAE-nCRP trade-off is the direct dependency modeling among clusters against the mean-field variational approach" that don't convey their intended meaning (at least to me).
* The random variable η seems completely superfluous. It only affects the likelihood through the level indicator l, but the marginal distribution p(l) = ∫_η p(η, l)dη \propto α is tractable, since only one level is drawn per observation. (This is not the case for the traditional nCRP as used in topic modeling, since there a different level is chosen for each word.)
* The novelty over the nCRP-VAE approach of Goyal et al. (2017) is pretty minor. The main difference seems to be that the model can select clusters at different levels, but I didn't quite get the intuition for why this should be desirable. In topic modeling, higher-level clusters tend to contain less-specialized words, and each document is a mix of specialized and general topics. But in this model, only one level is used to explain an entire image or document, and the idea that an entire image or document is much "more specialized" than another doesn't seem very intuitive to me.


VARIANCE NETWORKS: WHEN EXPECTATION DOES NOT MEET YOUR EXPECTATIONS
Ordinary stochastic neural networks mostly rely on the expected values of their weights to make predictions, whereas the induced noise is mostly used to capture the uncertainty, prevent overfitting and slightly boost the performance through test-time averaging. In this paper, we introduce variance layers, a different kind of stochastic layers. Each weight of a variance layer follows a zero-mean distribution and is only parameterized by its variance. It means that each object is represented by a zero-mean distribution in the space of the activations. We show that such layers can learn surprisingly well, can serve as an efficient exploration tool in reinforcement learning tasks and provide a decent defense against adversarial attacks. We also show that a number of conventional Bayesian neural networks naturally converge to such zero-mean posteriors. We observe that in these cases such zero-mean parameterization leads to a much better training objective than more flexible conventional parameterizations where the mean is being learned.

This paper studies variance neural networks, which approximate the posterior of Bayesian neural networks with zero-mean Gaussian distributions. The inference results are surprisingly well though there is no information in the mean of the posterior. It further shows that the several variational dropout methods are closed related to the proposed method. The experiment indicates that the ELBO can actually better optimized with this restricted form of variational distribution.
The paper is clearly written and easy to follow. The technique in the paper is solid.
However, the authors might need to clarify a few questions below.
Q1:  if every transformation is antisymmetric non-linear, then it seems that the expected distribution of $t$ in (2) is zero. Is this true or not? In another word, class information has to be read out from the encoding of instances in Fig 1. It seems antisymmetric operators cannot do so, as it will only get symmetric distributions from symmetric distributions.
Q2: it is not straightforward to see why KL term needs to go zero. In my understanding, the posterior aims to fit two objectives: maximizing data likelihood and minimizing KL term. When the signal from the data is strong (e.g. large amount of data), the first objective becomes more important. Then q does not really try to make KL zero, and alpha has no reason to go infinity. Can you explain more?
Q3: Is the claimed benefit from the optimization procedure or the special structure of the variance layer? Is it possible to test the hypothesis by 1) initializing a q distribution with learnable mean by the solution of variance neural network and then 2) optimizing q? Then the optimization procedure should continue to increase ELBO. Then compare the learned q against the variance neural network. If the learned q is better than the variance network -- it means the network structure is better for optimization, but the structure itself might not be so special. If the learned q is worse than the variance network, then the structure is interesting.
A few detailed comments:
1. logU used without definition.
2. if the paper has a few sentence explaining "Gaussian dropout approximate posterior", section 4 will be smoother to read.


DEEP, SKINNY NEURAL NETWORKS ARE NOT UNI- VERSAL APPROXIMATORS
In order to choose a neural network architecture that will be effective for a particular modeling problem, one must understand the limitations imposed by each of the potential options. These limitations are typically described in terms of information theoretic bounds, or by comparing the relative complexity needed to approximate example functions between different architectures. In this paper, we examine the topological constraints that the architecture of a neural network imposes on the level sets of all the functions that it is able to approximate. This approach is novel for both the nature of the limitations and the fact that they are independent of network depth for a broad family of activation functions.

This is a very nice paper contributing to what I consider a relatively underexplored but potentially very promising research direction. The title of the paper in my opinion undersells the result which is not only that "deep skinny neural networks" are not universal approximators, but that the class of functions which cannot be approximated includes a set of practically relevant classifiers as illustrated by the figure on page 8. The presentation is extremely clear with helpful illustrations and toy but insightful experiments.
My current rating of this paper is based on assuming that the following concerns will be addressed. I will adjust the score accordingly after authors' reply.
Main:
- A very similar result can be found in Theorem 7 of Beise et al.'s "On decision regions of narrow deep neural networks" from July 2018 ( https://arxiv.org/abs/1807.01194 )
	Some differences:
		- The other paper considers connected whereas this paper considers path-connected components (the former is more general).
		- The other paper only considers multi-label classification, this paper is relevant to all classification and regression problems (the latter is more general).
		- The other paper requires that the activation function is "strictly monotonic or ReLU" whereas this paper allows "uniformly approximable with one-to-one functions" activations (the latter is more general).
	The result in this paper seems slightly more general but largely similar. Can you please comment on the differences/relation to the other paper?
- Proof of Lemma 4:  "Thus the composition \hat{f} is also one-to-one, and therefore a homeomorphism from R^n onto its image I_{\hat{f}}". Is it not necessary that \hat{f} has a continuous inverse in order to be a homeomorphism? I do not immediately see whether the class of activation functions considered in this paper implies that this condition is satisfied. Please clarify.
Minor:
- Proof of Lemma 5: It seems g is assumed to be continuous at several places (e.g. "... level sets of are closed as subsets of R^n ..." seems to assume that pre-image of a closed set under g is closed, or later "This implies g(F) is a compact subset of R ..."). Perhaps you are assuming that M is a set of continuous functions and using the fact that uniform limit of continuous functions is continuous? Please clarify.
- On p.4: "This is fairly immediate from the assumptions on \varphi and the fact that singular transition matrices can be approximated by non-singular ones." Is the second part of the sentence using the assumption that the input space is compact? Please clarify.
- Second line in Section 5: i < k should probably be i < \kappa.


This paper extends the proof of density of neural networks in the space of continuous (or even measurable) functions on Euclidean spaces to functions on compact sets of probability measures. By doing so the work parallels a more then a decade old results on mean-map embedding of probability measures in reproducing kernel Hilbert spaces. The work has wide practical consequences for multi-instance learning, where it theoretically justifies some recently proposed constructions. The result is then extended to Cartesian products, yielding universal approximation theorem for tree-structured domains, which naturally occur in data-exchange formats like JSON, XML, YAML, AVRO, and ProtoBuffer. This has important practical implications, as it enables to automatically create an architecture of neural networks for processing structured data (AutoML paradigms), as demonstrated by an accompanied library for JSON format.

The authors study in this paper the approximation capabilities of neural networks for real valued functions on probability measure spaces (and on tree structured domains).
The first step of the paper consists in extending standard NN results to probability measure spaces, that is rather than having finite dimensional vectors as inputs, the NN considered here have probability measures as inputs. The extension to this case is straightforward and closely related to older extension on infinite dimensional spaces (see for instance the seminal paper of Stinchcombe https://doi.org/10.1016/S0893-6080(98)00108-7 and e.g. http://dx.doi.org/10.1016/j.neunet.2004.07.001 for an application to NN with functional inputs). Nothing quite new here.
In addition, and exactly as in the case of functional inputs, the real world neural networks do not implement what is covered by the theorem but only an approximation of it. This is acknowledged by the authors at the end of Section 2 but in a way that is close to hand waving. Indeed while the probability distribution point is valuable and gives interesting tools in the MIL context, the truth is that we have no reason to assume the bag sizes will grow to infinite or even will be under our control. In fact there are many situations were the bag sizes are part of the data (for instance when a text is embedded in a vector space word by word and then represented as a bag of vectors). Thus proving some form of universal approximation in the multiple instance learning context would need to take this fact into account, something that is not done at all here.
Therefore I believe the contribution of this paper to be somewhat limited.


CONVOLUTIONAL NEURAL NETWORKS ON NON- UNIFORM GEOMETRICAL SIGNALS USING EUCLIDEAN SPECTRAL TRANSFORMATION
Convolutional Neural Networks (CNN) have been successful in processing data signals that are uniformly sampled in the spatial domain (e.g., images). However, most data signals do not natively exist on a grid, and in the process of being sampled onto a uniform physical grid suffer significant aliasing error and information loss. Moreover, signals can exist in different topological structures as, for example, points, lines, surfaces and volumes. It has been challenging to analyze signals with mixed topologies (for example, point cloud with surface mesh). To this end, we develop mathematical formulations for Non-Uniform Fourier Transforms (NUFT) to directly, and optimally, sample nonuniform data signals of different topologies defined on a simplex mesh into the spectral domain with no spatial sampling error. The spectral transform is performed in the Euclidean space, which removes the translation ambiguity from works on the graph spectrum. Our representation has four distinct advantages: (1) the process causes no spatial sampling error during the initial sampling, (2) the generality of this approach provides a unified framework for using CNNs to analyze signals of mixed topologies, (3) it allows us to leverage state-of-the-art backbone CNN architectures for effective learning without having to design a particular architecture for a particular data structure in an ad-hoc fashion, and (4) the representation allows weighted meshes where each element has a different weight (i.e., texture) indicating local properties. We achieve results on par with the state-of-the-art for the 3D shape retrieval task, and a new state-of-the-art for the point cloud to surface reconstruction task.

Overall Thoughts:
I think this paper addresses an interesting topic and that the community as a whole are interested on the application of learning algorithms to non-Euclidean domains. It is nice to see the application of Fourier sampling to geometric primitives in a sensible manner and I am positive about that part of the paper. However, in its current form, I have quite a few questions about the approach and the empirical studies - I would need to here more information from the authors on the points below.
Specific Comments/Questions:
Sec1: The authors make a number of assertions about the representational errors that occur in other approaches - I feel that these claims should be supported by specific references.
Contributions: It is not clear to me that the experiments show that the method “preserves maximal information content” - in my understanding, information content has a formal definition and I don’t see where this is presented in the results?
Sec2: Before CNNs there has been a substantial analysis of Fourier methods applied to shape models, e.g. elliptical Fourier series for shape contours (and signed distance representations) by Prisacariu et al.
Sec2: It is also worth noting that there is substantial literature on non-uniform Fourier methods including the non-uniform Fourier transform and a number of accelerations (e.g. NUFFT) as well as consideration of the implications of band-limiting the sampled spectra.
Sec3: The mathematical derivation all makes sense to me and makes use of results for piecewise uniform signals. Please could the authors provide more details on how the spectra are represented? These discontinuous signals (esp. delta functions) will have infinite bandwidth in the spectral domain so how they are stored would seem very important to me. Are the signals band limited at some point? If so, how does this affect the approximation and should filtering/windowing be used? Otherwise is there not a difficult storage problem? The final paragraph suggests all the analytic signals might be stored but this has a big impact on how efficient the algorithm is and really is far too important a point to just have a single sentence - please can the authors expand on how this is actually implemented and what the computational considerations are (and resulting impacts on performance)?
Sec4: Please can the authors add error bars (at the least) to all the tables/plots in the results. It is entirely unreasonable to make any statements about how significant the results may be without even the most basic of analysis. Ideally we should see histograms of the results for the retrieval and shape reconstruction results.
Sec4: How is the downsampling performed in the MNIST experiment? It would seem very important to take care with this for the purpose of comparison. A significant disparity between the signed distance function and the NUFT would seem slightly surprising to me? Again, without error bars we really cannot say very much about the results on the right of Fig3(b).
Sec4: Could the Fig4 results not be provided as histograms? We also need many more details about how the results were obtained and procedures to ensure that the results are meaningful and robust (e.g. repeated tests and partitions of the data, etc..)
Arch and Training Details: How are we to know that these choices provide fair comparisons to previous approaches?
Sec3/4: All the results seem to require the input to be reconstructed back into a dense sampling domain (via inverse FT) - is this the case? Would it not be more efficient to perform the convolutions in the spectral domain where the signal is sparse?
Sec4: It seems pretty unfair to train and test on a single category of shapes in the shape test since the data is known not to be very diverse? Particularly when, unless I’ve misunderstood, the baselines on the shape recovery test to not involve learning and so (while helpful to have) they are not really fair baselines compared to other learning approaches? Also, please can the authors provide much more information about the extra processing applied (e.g. the part starting with the “extra mesh thickness”) since there seem to be some extra steps that are nothing to do with the rest of the method and may impact the results significantly.
Sec4: It is interesting that Table 3 (again difficult to say without error bars) indicates that there are times when the method performs better with the additional of noise - this seems counter-intuitive - please could the authors comment on this?
Other Points:
I’m afraid that there are quite a few grammatical errors in the text (too many to list here) so I would recommend another round of proof-reading.


Transferring learned knowledge from one environment to another is an important step towards practical reinforcement learning (RL). In this paper, we investigate the problem of transfer learning across environments with different dynamics while accomplishing the same task in the continuous control domain. We start by illustrating the limitations of policy-centric methods (policy gradient, actorcritic, etc.) when transferring knowledge across environments. We then propose a general model-based value-centric (MVC) framework for continuous RL. MVC learns a dynamics approximator and a value approximator simultaneously in the source domain, and makes decision based on both of them. We evaluate MVC against popular baselines on 5 benchmark control tasks in a training from scratch setting and a transfer learning setting. Our experiments demonstrate MVC achieves comparable performance with the baselines when it is trained from scratch, while it significantly surpasses them when it is used in the transfer setting.

This paper proposes a model-based value-centric (MVC) framework for transfer learning in continuous RL problems, and an algorithm within that framework. The paper attempts to answer two questions: (1) "why are current RL algorithms so inefficient in transfer learning" and (2) "what kind of RL algorithms could be friendly to transfer learning by nature"? I think these are very interesting questions to investigate, and researchers that work on transfer learning could benefit from insights on them. However, I am not yet convinced that this paper answers these questions satisfyingly. It would be great to hear the author's thoughts on my questions below.
The main insight I take away from the paper is that policy gradient methods are not suitable for transfer learning compared to model-based and value-centric methods for some assumptions (the reward function not changing and the transition dynamics being deterministic). This insight and the experiments in the paper are interesting, but I am unsure if the paper as it is presented now passes the bar for ICLR.
In general the paper has two contributions:
A) analysis of value-centric vs policy-centric methods
B) an algorithm that is more useful for transfer learning.
Regarding A)
The authors argue that policy-centric algorithms are less useful for transfer learning than value-centric methods.
They first illustrate this with an example in Section 3. Since this is just one example, as a reader I wonder if it would not be possible to construct an example that shows the exact opposite, where value iteration fails but policy gradient doesn't. It feels like there are many assumptions that play into the given example (the reward function not changing; the transition dynamics being deterministic; the choice of using policy gradients and value iteration).
In addition, the authors provide a theoretical justification in the Appendix (which I have briefly scanned) and the intuition behind it in Section 5. From what I understand, the main problem arises from the policy's output space being a Gaussian distribution, which causes the policy being able to get stuck in a local optimum. Further, the authors show (in the Appendix) that under some assumtions the value function always converges. Are there any guarantees on this when we don't have access to the true reward and transition functions (which themselves could get stuck in a local optimum)?
Would the authors say that the phenomenon is more a problem with the algorithm (policy gradient vs value iteration) than policy-centric and value-centric methods in general? Are there other methods that would be able to transfer policies better than policy gradient methods?
Regarding B)
The author's proposed method (MVC) has three components: the value function, the dynamics model and the reward model, all of which are learned by neural networks. It seems like the main advantage comes from using a model (since that's the aspect which changes when having to transfer to an altered MDP). Does the advantage of this method over DDPG and TRPO come from the fact that the dynamics model changes smoothly, and we have an approximation to it? Then it is not surprising that this outperforms a policy gradient method.
Other comments:
- Could you explain what is meant by "precise" and "imprecise" when speaking about policies or value functions?
- Could you explain what is meant by the algorithm being "accessible" (e.g., Definition 1)?
- Section 2.1: In Property 1, what is f? Could you make explicit why we are interested in the two properties listed? By "not rigorously", do you mean that those properties are based on intuition? These properties are used later in the paper and the appendix, so I wonder how strong of an assumption this is.
- Section 2.2: Could you explain what is meant by "task"? You say that within the MDP, the transition dynamics and reward functions change, but the task stays the same. However, earlier (in the introduction) you state that only the environment dynamics change. I find it confusing that "the task" is something hand-wavy and not part of the formal definition of the MDP. In what exact ways can the reward function be influenced by the change in the transition dynamics?
- Section 3: Replace "obviously" with "hence"; remove "it is not hard to find that". This might not be so trivial for some readers.
- Appendix B: Refer to Table 1 in the text.
Clarity: The paper is written well, but I think some assumptions and their affects should be stated more clearly and put into context. The paper misses a discussion / conclusion section. It would be great to see a discussion on some of the assumptions; e.g., what if the low dimensional assumtion breaks down? What if we assume that also the reward function can change? The authors are in a unique position to give insight into these things (even if the results from the paper do not hold after dropping some assumptions) and it would be very helpful to share these with the reader in a discussion section.


L-SHAPLEY AND C-SHAPLEY: EFFICIENT MODEL INTERPRETATION FOR STRUCTURED DATA
Instancewise feature scoring is a method for model interpretation, which yields, for each test instance, a vector of importance scores associated with features. Methods based on the Shapley score have been proposed as a fair way of computing feature attributions, but incur an exponential complexity in the number of features. This combinatorial explosion arises from the definition of Shapley value and prevents these methods from being scalable to large data sets and complex models. We focus on settings in which the data have a graph structure, and the contribution of features to the target variable is well-approximated by a graphstructured factorization. In such settings, we develop two algorithms with linear complexity for instancewise feature importance scoring on black-box models. We establish the relationship of our methods to the Shapley value and a closely related concept known as the Myerson value from cooperative game theory. We demonstrate on both language and image data that our algorithms compare favorably with other methods using both quantitative metrics and human evaluation.

This paper proposes two methods for instance-wise feature importance scoring, which is the task of ranking the importance of each feature in a particular example (in contrast to class-wise or overall feature importance).  The approach uses Shapely values, which are a principled way of measuring the contribution of a feature, and have been previously used in feature importance ranking.
The difficulty with Shapely values is they are extremely (exponentially) expensive to compute, and the contribution of this paper is to provide two efficient methods of computing approximate Shapely values when there is a known structure (a graph) relating the features to each other.
The paper first introduces the L(ocal)-Shapely value, which arises by restricting the Shapely value to a neighbourhood of the feature of interest.  The L-Shapely value is still expensive to compute for large neighbourhoods, but can be tractable for small neighbourhoods.
The second approximation is the C(onnected)-Shapely value, which further restricts the L-Shapely computation to only consider connected subgraphs of local neighbourhoods.  The justification for restricting to connected neighbourhoods is given through a connection to the Myerson value, which is somewhat obscure to me, since I am not familiar with the relevant literature.  Nonetheless, it is clear that for the graphs of interest in this paper (chains and lattices) restricting to connected neighbourhoods is a substantial savings.
I have understood the scores presented in Figures 2 and 3 as follows:
For each feature of each example, rank the features according to importance, using the plugin estimate for P(Y|X_S) where needed.
For each "percent of features masked" compute log(P(y_true | x_{S\top features})) - log(P(y_true | x)) using the plugin estimate, and average these values over the dataset.
Based on this understanding the results are quite good.  The approximate Shapely values do a much better job than their competitors of identifying highly relevant features based on this measure.  The qualitative results are also quite compelling, especially on images where C-Shapely tends to select contiguous regions which is intuitively correct behavior.
Comparing the different methods in Figure 4, there is quite some variability in the features selected by using different estimators of Shapley values.  I wonder is there some way to attack the problem of distinguishing when a feature is ranked highly when its (exact) Shapley value is high versus when it is ranked highly as an artifact of the estimator?


Deep learning on an edge device requires energy efficient operation due to ever diminishing power budget. Intentional low quality data during the data acquisition for longer battery life, and natural noise from the low cost sensor degrade the quality of target output which hinders adoption of deep learning on an edge device.
To overcome these problems, we propose simple yet efficient mixture of preprocessing experts (MoPE) model to handle various image distortions including low resolution and noisy images. We also propose to use adversarially trained auto encoder as a pre-processing expert for the noisy images. We evaluate our proposed method for various machine learning tasks including object detection on MS-COCO 2014 dataset, multiple object tracking problem on MOT-Challenge dataset, and human activity classification on UCF 101 dataset. Experimental results show that the proposed method achieves better detection, tracking and activity classification accuracies under noise without sacrificing accuracies for the clean images. The overheads of our proposed MoPE are 0.67% and 0.17% in terms of memory and computation compared to the baseline object detection network.

Summary
This paper introduced a parameterized image processing technique to improve a robustness of visual recognition systems against noisy input data. The proposed method is composed of two components; a denoising network that suppresses the noise signals in an image, and gating network that predicts whether to use the original input image or the one produced by the denoising network. The proposed idea is evaluated on three tasks of object detection, tracking and action recognition.
Originality and significance:
The originality of the paper is very limited since the paper simply combines the existing image denoising technique with the idea of gating. The practical significance of the work is also limited since the model is trained and evaluated with only synthetically generated noise patterns; it is not surprising that the proposed method (both denoising and gating networks) works under this setting, as the noise is created synthetically under the same setting in both training and testing. To demonstrate the practical usefulness, it would be great if the model is evaluated with the actual source of noises (e.g. noises from input sensors, distortion by image compression, etc).
Clarity:
I think the title of the paper is misleading; the proposed model is actually not a mixture of preprocessing units, as it combines *a* denoising unit together with identity mapping. The gating network is also not designed to incorporate a mixture of more than two preprocessing units, as it outputs only “on/off switches” instead of weights for K mixture components (K>2).
Minor comments:
1) the paper argued the importance of lightweight preprocessing but have not provided analysis on computation costs. From the current results, I don’t see the clear benefit of the proposed method (denoising network) over the average filtering considering the tradeoff between computation vs. performance.
2) In Figure 5, I suggest highlighting the differences among the examples for clarity.


One popular hypothesis of neural network generalization is that the flat local minima of loss surface in parameter space leads to good generalization. However, we demonstrate that loss surface in parameter space has no obvious relationship with generalization, especially under adversarial settings. Through visualizing decision surfaces in both parameter space and input space, we instead show that the geometry property of decision surface in input space correlates well with the adversarial robustness. We then propose an adversarial robustness indicator, which can evaluate a neural network's intrinsic robustness property without testing its accuracy under adversarial attacks. Guided by it, we further propose our robust training method. Without involving adversarial training, our method could enhance network's intrinsic adversarial robustness against various adversarial attacks.

This paper argues that analyzing loss surfaces in parameter space for the purposes of evaluating adversarial robustness and generalization is ineffective, while measuring input loss surfaces is more accurate. By converting loss surfaces to decision surfaces (which denote the difference between the max and second highest logit), the authors show that all adversarial attack methods appear similar wrt the decision surface. This result is then related to the statistics of the input Jacobian and Hessian, which are shown to differ across adversarially sensitive and robust models. Finally, a regularization method based on regularizing the input Jacobian is proposed and evaluated. All of these results are shown through experiments on MNIST and CIFAR-10.
In general, the paper is clear, though there are a number of typos. With respect to novelty, some of the experiments are novel, but others, including the improved training method, have been explored before (see specific comments for references). Finally, regarding significance, many of the insights provided this paper are true by definition, and are therefore unlikely to have a significant impact.
While I strongly believe that rigorous empirical studies of neural networks are essential, this paper is lacking in several key areas, including framing, experimental insights, and relation to prior work, and is therefore difficult to recommend. Please see the comments below for more detail.
Major comments:
1) In the beginning of the paper, adversarial robustness and generalization are equated. However, adversarial robustness and generalization are not necessarily equivalent, and in fact, several papers have provided evidence against this notion, showing that adversarial inputs are likely to be present even for very good models [5, 6] and that adversarially sensitive models can often generalize quite well [2]. Moreover, all the experiments within the paper only address adversarial robustness rather than generalization to unperturbed samples.
2) One of the main results of this paper is that the loss surface wrt input space is more sensitive to adversarial perturbations than the loss surface wrt parameter space. Because adversarial inputs are defined in input space, by definition, the loss surface wrt to the input must be sensitive to adversarial examples. This result therefore appears true by definition. Moreover, [3] related the input Jacobian to generalization, finding a similar result, but is not discussed or cited.
3) The main result of Section 3 is that all adversarial attacks “utilize the decision surface geometry properties to cross the decision boundary within least distance.” While to my knowledge the decision surface visualization is novel and might have important uses, this statement is again true by definition, given that adversarial attack methods try to find the smallest perturbation which changes the network decision. As a result, all methods must find directions which are short paths in the decision surface. It is therefore unclear what additional insight this analysis presents.
4) How does measuring the loss landscape as an indicator for adversarial robustness differ from simply trying to find adversarial examples as is common? If anything, it seems it should be more computationally expensive as points are sampled in a grid search vs optimized for.
5) The proposed regularizer for adversarial robustness, based on regularizing the input Jacobian, is very similar to what was proposed in [1], yet [1] is not discussed or cited.
Minor comments:
1) The paper’s first sentence states that “It is commonly believed that a neural network’s generalization is correlated to ...the flatness of the local minima in parameter space.” However, [4] showed several years ago that the local minima flatness can be arbitrarily rescaled and has been fairly influential. While [4] is cited in the paper, it is only cited in the related work section as support for the statement that local minima flatness is related to generalization when this is precisely opposite the point this paper makes. [4] should be discussed in more detail, both in the introduction and the related work section.
2) The paper is quite lengthy, going right up against the hard 10 page limit. While this may be acceptable for papers with large figures or which require the extra space, this paper does not currently meet that threshold.
3) Throughout the figures, axes should be labeled.
4) In section 2.2, it is stated that both networks achieve optimal accuracy of ~90% on CIFAR-10. This is not optimal accuracy and hasn’t been for several years [7].
5) Why is equation 2 calculated with respect to the logit layer vs the normalized softmax layer? Using the unnormalized logits may introduce noise due to scaling.
6) In Figure 8, the scales of the Hessian are extremely different. Does this impact the measurement of sparseness?
Typos:
1) Introduction, second paragraph: “For example, ResNet model usually converges to…” should be “For example, ResNet models usually converge to…”
2) Introduction, second paragraph: “...defected by the adversarial noises...” should be “...defected by adversarial noise…”
3) Introduction, third paragraph: “...introduced by adversarial noises...” should be “...introduced by adversarial noise…”
4) Section 3.1, first paragraph: “cross entropy based loss surface is…” should be “cross entropy based loss surfaces is…”
[1] Jakubovitz, Daniel, and Raja Giryes. "Improving DNN Robustness to Adversarial Attacks using Jacobian Regularization." arXiv preprint arXiv:1803.08680 (2018). ECCV 2018.
[2] Zahavy, Tom, et al. "Ensemble Robustness and Generalization of Stochastic Deep Learning Algorithms." arXiv preprint arXiv:1602.02389 (2016). ICLR Workshop 2018
[3] Novak, Roman, et al. "Sensitivity and generalization in neural networks: an empirical study." arXiv preprint arXiv:1802.08760 (2018). ICLR 2018.
[4] Dinh, Laurent, et al. "Sharp Minima Can Generalize For Deep Nets." International Conference on Machine Learning. 2017.
[5] Fawzi, Alhussein, Hamza Fawzi, and Omar Fawzi. "Adversarial vulnerability for any classifier." arXiv preprint arXiv:1802.08686 (2018). NIPS 2018.
[6] Gilmer, Justin, et al. "Adversarial spheres." arXiv preprint arXiv:1801.02774 (2018). ICLR Workshop 2018.
[7] http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html


Capsule Networks have shown encouraging results on defacto benchmark computer vision datasets such as MNIST, CIFAR and smallNORB. Although, they are yet to be tested on tasks where (1) the entities detected inherently have more complex internal representations and (2) there are very few instances per class to learn from and (3) where point-wise classification is not suitable. Hence, this paper carries out experiments on face verification in both controlled and uncontrolled settings that together address these points. In doing so we introduce Siamese Capsule Networks, a new variant that can be used for pairwise learning tasks. The model is trained using contrastive loss with 2 -normalized capsule encoded pose features. We find that Siamese Capsule Networks perform well against strong baselines on both pairwise learning datasets, yielding best results in the few-shot learning setting where image pairs in the test set contain unseen subjects.

This paper presents an extension of Capsule Networks, Siamese Capsule Networks (SCNs), that can be applied to the problem of face verification. Results are reported on the small AT&T dataset and the LFW dataset.
I like the direction that this paper is taking. The original Capsules work has been looking at fairly simple and small scale datasets, and the natural next step for this approach is to start addressing harder datasets, LFW being one of them. Also face verification is a natural problem to look at with Capsules.
However, I think this paper currently falls short of what I would expect from an ICLR paper. First, the results are not particularly impressive. Indeed, SCN doesn't outperform AlexNet on LFW (the most interesting dataset in the experiments). Also, I'm personally not particularly compelled by the use of the contrastive loss as the measure of performance, as it is sensitive to the scaling of the particular representation f(x) used to compute distances. Looking at accuracy (as in other face verification papers, such as DeepFace) for instance would have been more appropriate, in my opinion. I'm also worried about how hyper-parameters were selected. There are A LOT of hyper-parameters involved (loss function hyper-parameters, architecture hyper-parameters, optimizer hyper-parameters) and not much is said about how these were chosen. It is mentioned that cross validation was used to select some margin hyper-parameters, but results in Table 1 are also cross-validation results, which makes me wonder whether hyper-parameters were tuned on the performance reported in Table 1 (which of course would be biased).
The paper is also pretty hard to read. I recognize that there is a lot of complicated literature to cover (e.g. prior work on Capsule Networks has introduced variations on various aspects which are each complicated to describe). But as it currently reads, I can honestly say that I'm not 100% sure what exactly was implemented, i.e. which components of previous Capsule Networks were actually used in the experiments and which weren't. For example, I wasn't able to figure out which routing mechanism was used in this paper. The paper would strongly benefit from more explicitly laying out the exact definition of SCN, perhaps at the expense of enumerating all the other variants of capsules and losses that previous work has used.
Finally, regardless of the clarify of the paper, the novelty in extending Capsule Networks to a siamese architecture is arguably pretty incremental. This wouldn't be too much of a problem if the experimental results were strong, but unfortunately it isn't the case.
In summary:
Pros
- New extension of Capsule Networks, tackling a more challenging problem than previous work
Cons
- Novelty is incremental
- Paper lacks clarity and is hard to read
- Results are underwhelming
For these reasons, I'm afraid I can't recommend this paper be accepted.
Finally, I've noted the following typos:
- hinton1985shape => use proper reference
- within in => within
- that represent => that represents
- a Iterated => an Iterated
- is got => is obtained
- followed two => followed by two
- enocded => encoded
- a a pair => a pair
- such that to => such as to
- there 1680 subjects => there are 1680 subjects
- of varied amount => of the varied amount
- are used many => are used in many
- across the paper: lots of in-text references should be in parenthesis


Large-scale search is an essential task in modern information systems. Numerous learning based models are proposed to capture semantic level similarity measures for searching or ranking. However, these measures are usually complicated and beyond metric distances. As Approximate Nearest Neighbor Search (ANNS) techniques have specifications on metric distances, efficient searching by advanced measures is still an open question. In this paper, we formulate largescale search as a general task, Optimal Binary Functional Search (OBFS), which contains ANNS as special cases. We analyze existing OBFS methods' limitations and explain they are not applicable for complicated searching measures. We propose a flexible graph based solution for OBFS, Search on L2 Graph (SL2G). SL2G can achieve approximate optimal, with accessible conditions. Experiments demonstrate SL2G's efficiency in searching by advanced matching measures (i.e., Neural Network based measures).

Post-rebuttal
------------------
I have read the rebuttal and I better understand the paper. Given that, I am going to raise my rating by one point for the following reason:
- The manuscript presents a novel solution to a general problem and it is a valid solution. However, the solution is somewhat obvious, which is not necessarily a bad thing, which is why I am raising my rating by a point. However, an easy solution like the one proposed in the manuscript means that OBFS considered in this manuscript is not as general as the authors let on -- there is an implicit assumption that f(x_i, q) is close to f(x_j, q) if x_i is close to x_j.
- While the authors answered a lot of my clarification questions, the manuscript seems still a little hard to parse and can be significantly improved for easier reading and understanding.
=========================================
Pros
-------
[Originality/Significance] The manuscript focuses on a very general and important problem and proposes a scheme to solve this general problem. The authors present some theoretical and empirical results to demonstrate the utility of the proposed scheme.
Limitations
----------------
[Clarity] While the problem being addressing is extremely important, and the proposed solution seems reasonable, the manuscript is really hard to follow. For example, Definition 3 and Theorem 1 are extremely hard to understand.
[Clarity/Significance] Moreover, I feel that the authors should be more precise in pointing out why current graph based search algorithms are just not trivially applicable to OBFS. The nature of the approximate Delaunay graph is that it can be built for any given similarity function (the level of approximation obviously depends on the similarity function, but that is an existing issue with graph-based methods). Given the graph, I do not understand why the basic search algorithm on this similarity graph would not be an approximate solution to OBFS. Hence I believe the authors need to clarify why the existing graph based algorithms do not directly translate.
[Significance] While Definition 1 considers topological spaces, SL2G is assuming that X and (maybe) Y are in R^d (for different values of d). So does that mean that SL2G does not solve the general OBFS?
[Significance/Correctness/Clarity] The assumptions in Theorem 2 (as well as the supporting Proposition 1 in Appendix B) seems quite unreasonable. In moderately high dimensional X, doesn't the curse of dimensionality imply that this condition will not hold in most case? In there any reason why/how this would be circumvented? Moreover, in Proposition 1 (in Appendix B), the quantity C_r needs to be precisely defined since it could in general be exponential in the number of dimensions. Also, the assumption in Proposition 1 where \lambda^* > 0 is fairly strong in high dimensional data since data gets really sparse in high dimensions. Finally, the last step in Proposition 1 (where the failure probability obtained from the union bound is connected to condition (b) in Theorem 2) is not clear at all -- it is not apparent how E and F related to S and how p relates to every ball containing a point in S. This is a very important step and needs better exposition.
[Clarity/Significance] I am unable to understand the baseline HNSW-SBFG (or the motivation for it) in the empirical section. It would be good to clarify this.
General comments
---------------------------
[Significance] Finally, I believe that it would be good to see a connection between the success of SL2G to relationship between |f(x1, q) - f(x2, q)| and ||x1 - x2 ||_2 since the author emphasize that the proposed scheme can be seen as "gradient descent in Euclidean space" (although the authors would need to also precisely explain what they mean by that statement).
[Originality] Some related work that the authors should position their proposed problem/solution against:
- There is some work on "max-kernel search" which can perform similarity search with general notions of similarity (than just Euclidean metrics).
- There is some work on search with Bregman divergences which handle asymmetric similarity functions and also incorporate notions of gradient descent over convex sets.
Minor comments/typos
---------------------------------
- The authors should present the precise SL2G algorithm given the graph in the manuscript.
- l^2 --> \ell_2
- gradient decent --> gradient descent
- Table 1, f(q, x) --> f(x, q)


Recent theoretical work has demonstrated that deep neural networks have superior performance over shallow networks, but their training is more difficult, e.g., they suffer from the vanishing gradient problem. This problem can be typically resolved by the rectified linear unit (ReLU) activation. However, here we show that even for such activation, deep and narrow neural networks (NNs) will converge to erroneous mean or median states of the target function depending on the loss with high probability. Deep and narrow NNs are encountered in solving partial differential equations with high-order derivatives. We demonstrate this collapse of such NNs both numerically and theoretically, and provide estimates of the probability of collapse. We also construct a diagram of a safe region for designing NNs that avoid the collapse to erroneous states. Finally, we examine different ways of initialization and normalization that may avoid the collapse problem. Asymmetric initializations may reduce the probability of collapse but do not totally eliminate it.

In this paper, the authors investigate the collapse of deep and shallow network to a constant function under the following setting:
1. Very shallow networks, width ~ 10
2. ReLU activation function.
3. Symmetric weights and biases initialization.
4. Vanilla feed forward networks.
The main take-home message is: don't use neural networks (NNs) that are both deep and shallow.
The theoretical analysis is built on the observation:
1. Every neuron (after applying ReLU) is equal to zero with probability 1/2.
2. For narrow network and for any fixed input, there is a high chance that all neurons in a particular hidden layer are all zero. All neurons after that layer are all zero if zero-bias initialization is used.
3. The authors conclude that derivatives of all parameters vanish but the bias of the last layer.
4. As a result, the network collapse to its mean (median) if mean squared loss (L1 loss) is used because only the bias of the last layer is being updated during training.
Pros.
1. I think the phenomenon that shallow and deep NNs collapse to a constant is very interesting.
2. The authors provide empirical and theoretical insights in favor of wider networks: have a better chance to avoid vanishing gradients.
3. For shallow networks, it might be better not to use ReLU.
Cons:
1.The analysis works in a very limited setting, works for ReLU but not other activations: tanh, erf, SELU etc.
2. Very shallow networks are not popular in practice. Width>=100 is popular for fully-connected layers.
3. The phenomenon observed by the paper can easily be addressed using any of the following trick:
   3.1. Non-symmetric initialization  (set the mean to be non-zero).
   3.2 . wider networks.
4. Most of the analysis (and theorems)  are about one single input. In another word, distribution of the inputs have not been taken into account.
5. I don't think the author provides a completely rigorous justification for the collapse phenomenon.
Other comments.
1. Eq (2) in page 4 is not trivially correct. The expectation operator (w.r.p. to lower layers) is moved into the activation function phi, justification is needed for this step.
2. Theorem 4: when the Lebesgue measure of $\Omega$ is NOT finite, it is unclear how to define a uniform probability distribution on it.
3. Theorem 4: the integrability assumption on y should depend on the loss: for L2 loss (L1 loss), squared (absolutely) integrable  should be used. They are not the same.


LEARNING IMPLICITLY RECURRENT CNNS THROUGH PARAMETER SHARING
We introduce a parameter sharing scheme, in which different layers of a convolutional neural network (CNN) are defined by a learned linear combination of parameter tensors from a global bank of templates. Restricting the number of templates yields a flexible hybridization of traditional CNNs and recurrent networks. Compared to traditional CNNs, we demonstrate substantial parameter savings on standard image classification tasks, while maintaining accuracy.
Our simple parameter sharing scheme, though defined via soft weights, in practice often yields trained networks with near strict recurrent structure; with negligible side effects, they convert into networks with actual loops. Training these networks thus implicitly involves discovery of suitable recurrent architectures. Though considering only the design aspect of recurrent links, our trained networks achieve accuracy competitive with those built using state-of-the-art neural architecture search (NAS) procedures.
Our hybridization of recurrent and convolutional networks may also represent a beneficial architectural bias. Specifically, on synthetic tasks which are algorithmic in nature, our hybrid networks both train faster and extrapolate better to test examples outside the span of the training set.

The manuscript introduces a novel and interesting approach to weight sharing among CNNs layers, by learning linear combinations of shared weight templates. This allows parameter reduction, better sample efficiency. Furthermore, the authors propose a very simple way to inspect which layers choose similar combinations of template, as well as to push the network toward using similar combinations at each layer. This regularization term has a clear potential for computation reuse on dedicated hardware. The paper is well written, the method is interesting, the results are convincing and thoroughly conducted. I recommend acceptance.
1) It would be interesting to explore how often the layer parameters converge to similar weights and how similar. To this end I suggest to plot a 2d heatmap representing the similarity matrices between every pair of layers.
2) Figure 1 is not of immediate interpretability. Especially for the middle figure, what does the dotted box represent? What is the difference between weights and templates? Also it’s unclear which of the three options corresponds to the proposed method. I would have thought the middle one, but the text seems to indicate it is the rightmost one instead.
3) How are the alphas initialized? How fast are their transitions? Do they change smoothly over training? Do they evolve rapidly and plateau to a fixed value or keep changing during training? It would be really interesting to plot their value and discuss their evolution.
4) While the number of learned parameters is indeed reduced when the templates are shared among layers - which could lead to better sample efficiency - I am not sure whether the memory footprint on GPU would change (i.e., I believe that current frameworks would allocate the same kernel n-times if the same template was shared by n layers, but I am not certain). Although the potential reduction of the number of trainable parameters is an important result by itself, I wonder if what you propose would also allow to run bigger models on the same device or not, without heavy modifications of the inner machineries of pyTorch or Tensorflow.  Can you comment on this? Also note that the soft sharing regularization scheme that you propose can be of great interest for FPGA hardware implementations, that benefit a lot from module reuse. You could mention that in the paper.
5) Sec 4.1, the number of layers in one group is defined as (L-4)/3. It’s unclear to me where the 4 comes from. Also on page 6, k = (L-2)/3 - 2 is said to set one template per layer. I thought the two formulas would be the same in that case. What am I missing? Is it possible that one of the two formulas contain a typo (I believe that at the very least it should be either (L-2) or (L-4) in both cases)?
6) Sec 4.1, I find the notation SWRN-L-w-k and SWRN-L-w confusing. My suggestion is to set k to be the total number of templates (as opposed to the number of templates *per group of layers*), which makes it much easier to relate to, and most importantly allows for an immediate comparison with the capacity of the vanilla model. As a side effect, it also makes it very easy to spot the configuration with one template per layer (SWRN-L-w-L) thus eliminating the need for an ad-hoc notation to distinguish it.
7) The authors inspect how similar the template combination weights alphas are among layers. It would also be interesting to look into what is learned in the templates. CNN layers are known to learn peculiar and somewhat interpretable template matching filters. It would be really interesting to compare the filters learned by a vanilla network and its template sharing alternative. Also, I would welcome an analysis of which templates gets chosen the most at each layer in the hierarchy. It would be compelling if some kind of pattern of reuse emerged from learning.
8) Sec 4.4, it is unclear to me what can be the contribution of the 1x1 initial convolution, since it will see no context and all the information at the pixel level can be represented by a binary bit. Also, are the 3x3 convolutions “same” convolutions? If not, how are the last feature maps upscaled to be able to predict at the original resolution?
9) At the end of sec 4.4 the authors claim that the SCNN is “also advantaged over a more RNN-like model”. I fail to understand how to process this sentence, but I have a feeling that it’s incorrect to make any claims to the performance of “RNN-like models” as such a model was not used as a baseline in the experiments in any way. Similarly, in the conclusions I find it a bit stretched to claim that you can “gain a more flexible form of behavior typically attributed to RNNs”. While it’s true that the proposed network can in theory learn to reuse the same combination of templates, which can be mapped to a network with recursion, the results in this direction don’t seem strong enough to draw any conclusion and a more in-depth comparison against RNN performance would be in order before making any claim in this direction.
MINOR
- Sec3: I wouldn’t say the parameters are shared among layers in LSTMs, but rather among time unrolls.
- One drawback of the proposed method is that the layers are constrained to have the same shape. This is not a major disadvantage, but is still a constraint that would be good to make more explicit in the description of the model.
- Sec3, end of page 3: does the network reach the same accuracy as the vanilla model when k=L? Also, does the network use all the templates? How is the distribution of the alpha weights across layers in this case?
- Sec3.1, the V notation makes the narrative unnecessarily heavy. I suggest to drop it and refer directly to the templates T. Also the second part of the section, with examples of templates, doesn’t add much in my opinion and would be better depicted with a figure.
- Sec3.1, the e^(i) notation can be confused with an exp. I suggest to replace it with the much more common 1_{i=j}.
- Figure 2 depicts the relation between the LSM matrix and the topology of the network. This should be declared more clearly in the caption, in place of the ambiguous “capturing implicit recurrencies”. Also, the caption should explain what black/white stand for as well, and possibly quickly describe what the LSM matrix is. Also, it would be more clear that the network in the middle is equivalent to that on the right if the two were somehow connected in the figure. To this end they could, e.g., share a single LSM matrix among them. Finally, if possible try and put the LSM matrices on top of the related network, so that it’s clear which network they refer to. Sec 3.2 should also refer to Fig2 I believe.
- Table 1: I suggest to leave the comment on the results out of the caption, since it’s already in the main text.
- Table 2: rather than using blue, I suggest to underline the overall best results, so that it’s visible even if the paper is printed in B&W.
- Fig 3, I would specify that it’s better viewed in color
- Discussion: I feel the discussion of Table 1 is a bit difficult to follow. It could be made easier by reporting the difference in test error against the corresponding vanilla model (e.g., “improves the error rate on CIFAR10 by 0.26%”, rather than reporting the performance of both models)
- Fig 4, are all the stages the same and is the network in the left one such stages? If so, update the caption to make it clear please.
- Fig 4, which lambda has been used? Is it the same for all stages?
- Fig 5, specify that the one on the right is the target grid. Also, I believe that merging the two figures would make it easier to understand (e.g., some of the structure in the target comes from how the obstacles are placed, which requires to move back and forth from input to target several times to understand)
- Sec 4.4, space permitting, I would like to see at least one sample of what kind of shortest path prediction the network can come up with.
A few typos:
    * End of 3.2: the closer elements -> the closer the elements
    * Parameter efficiency: the period before re-parametrizing should probably be a comma?
    * Fig 4, illustration of stages -> illustration of the stages
    * End of pag7, an syntetic -> a syntetic


Learning to imitate expert actions given demonstrations containing image observations is a difficult problem in robotic control. The key challenge is generalizing behavior to out-of-distribution states that differ from those in the demonstrations. State-of-the-art imitation learning algorithms perform well in environments with low-dimensional observations, but typically involve adversarial optimization procedures, which can be difficult to use with high-dimensional image observations. We propose a remarkably simple alternative based on off-policy soft Q-learning, which we call soft Q imitation learning (SQIL, pronounced "skill"), that rewards the agent for matching demonstrated actions in demonstrated states. The key idea is initially filling the agent's experience replay buffer with demonstrations, where rewards are set to a positive constant, and setting rewards to zero in all additional experiences. We derive SQIL from first principles as a method for performing approximate inference under the MaxCausalEnt model of expert behavior. The approximate inference objective trades off between a pure behavioral cloning loss and a regularization term that incorporates information about state transitions via the soft Bellman error. Our experiments show that SQIL matches the state of the art in low-dimensional environments, and significantly outperforms prior work in playing video games from high-dimensional images.

The paper introduces a relatively simple method for imitation learning that seems to be successful despite its simplicity. The method, SQIL, assigns a constant positive reward (r) to the demonstrations and zero reward to generated trajectories. While I like the connections between SQIL and SQL and the simplicity of the idea, I think there are several issues which connections with GAIL that are not discussed; some "standard" environments (such as Mujoco) that SQIL has not compared against the baselines. I believe the paper would have a bigger impact after some addressing some of the issues.
(
Update: I am glad that the authors added updates to the experiments. I think the method could be practical due to the simplicity, therefore of interest to ICLR.
The Pong case is also quite interesting, although it seems slightly "unfair" since the true reward of Pong is also sparse and DQN could do well on it. I think the problem with GAIL is that the reward could be hard to learn in high-dimensional cases, so it is hard to find good hyperparameters for GAIL on the Pong case. This shows some potential of the idea behind using simple rewards.
)
1. The first issue is the similarity with GAIL in the "realistic" setting. Since we cannot have infinite expert demonstrations, there would always be some large enough network that could perfectly distinguish the demonstrations (assign reward to 1) and the generated policies (assign reward to 0). Therefore, it would seem to me that from this perspective SQIL is an instance of GAIL where the discriminator is powerful and expert demos are finite (and disjoint from generated trajectories, which is almost always the case for continuous control). In the finite capacity case, I am unsure whether the V and Q networks in SQIL does a similar job as the discriminator in GAIL / AIRL type algorithms, since both seem to extrapolate between demonstrations and generations?
2. Moreover, I don't think SQIL would always recover the expert policy even with infinite demonstrations. For example, lets think about the Reacher environment, where the agent controls a robotic arm to reach a target location. The expert demonstration is the fastest way of reaching the target (move minimum distance between joints). If we consider the MDP to have possibly very large / infinite horizon (only stops when it reaches the target), I could construct a hack policy that produces larger episodic reward compared to the expert. The policy would simply move back and forth between two expert demonstrated states, where it would receive 1 reward in the states for odd time and 0 reward for the states for even time. The reward would be something like 1 / (1 - \gamma^2) compared to the experts' reward which is \sum_{i=0..T} \gamma^{i} = (1 - \gamma^{T+1}) / (1 - \gamma).
Some fix would be to set the reward for generated policies to be negative, or introduce some absorbing state where the expert will still receive the positive reward even after reaching the target (but that is not included in demonstrations). Nevertheless, a suitable reward prior seems to be crucial to the success of this SQIL, as with GAIL requiring reward augmentation.
3. Despite the above issues, I think this could be a very practical method due to its (perhaps surprising) simplicity compared to GAIL. However, the experiments only considered two environments that are not typically considered by GAIL; I believe SQIL would make a bigger impact if it is compared with GAIL in Mujoco environments -- seems not very implementation heavy because your code is based on OpenAI baselines anyway. Mujoco with image inputs would also be relevant (see ACKTR paper).
Minor points:
- What is the underlying RL algorithm for GAIL? It would seem weird if you use Q-learning for SQIL and TRPO for GAIL, which makes it impossible to identify whether Q-learning or SQIL contributed more to the performance. While GAIL used TRPO in the original paper, it would be relatively straightforward to come up with some version of GAIL that uses Q-learning.
- Some more details in background for MaxEnt RL to make the paper more self contained.
- More details about the hyperparameters of SQIL in experiments -- e.g. what is \lambda?
- Did you pretrain the SQIL / GAIL policies? Either case, it would be important to mention that and be fair in terms of the comparison.
- Why does SQIL-11 perform worse than SQIL even though it is a tighter bound?
- wrt. math -- I think the anonymous comment addressed some of my concerns, but I have not read the updated version so cannot be sure.