GO GRADIENT FOR EXPECTATION-BASED OBJECTIVES
Within many machine learning algorithms, a fundamental problem concerns efficient calculation of an unbiased gradient wrt parameters γ for expectation-based objectives E qγ (y) [f (y)]. Most existing methods either (i) suffer from high variance, seeking help from (often) complicated variance-reduction techniques; or (ii) they only apply to reparameterizable continuous random variables and employ a reparameterization trick. To address these limitations, we propose a General and One-sample (GO) gradient that (i) applies to many distributions associated with non-reparameterizable continuous or discrete random variables, and (ii) has the same low-variance as the reparameterization trick. We find that the GO gradient often works well in practice based on only one Monte Carlo sample (although one can of course use more samples if desired). Alongside the GO gradient, we develop a means of propagating the chain rule through distributions, yielding statistical back-propagation, coupling neural networks to common random variables.

* Summary
The paper proposes an improved method for computing derivatives of the expectation. Such problems arises with many probabilistic models with noises or latent variables. The paper proposes a new gradient estimator of low variance applicable in certain scenarios, in particular it allows training of generative models in which observations and/or latent variables are discrete.
The submission clearly improves the state-of-the-art, experimentally demonstrates the method on several problems comparing with the alternative techniques. In what concerns the optimization, the method achieves a better objective value much faster, confirming that it is a lower variance gradient estimator.
The clarity of the presentation (in particular the description of when the method is applicable) and the technical correctness of the paper are somewhat lacking. In terms of applicability, it seems that many cases where discrete latent variables would be really interesting are not covered (e.g. sigmoid belief networks); the paper demonstrates experiments with discrete images (binary or 4-bit) not particularly motivated in my opinion. It also contains lots of additional technical details and experiments in the appendix, which I unfortunately did not review.
* Clarity
In the abstract the paper promises more than it delivers. Many problems can be cast as optimizing an expectation-based objective. The result does not at all apply to all of them. The reparameterization trick does not apply to all continuous random variables, only to such that the reparameterization satisfies certain smoothness conditions. Discrete variables are supported by the method only in the case that the distribution factors over all discrete variables conditionally on any additional “continuous variables” (to which the reparameterization trick is applicable). This very much limits the utility of the method. In particular it is not applicable to learning e.g. sigmoid belief networks [Neal, 92] (with conditional Bernoulli units) and many other problems.
“reparametrizable distributions”
A Bernoulli(p) random variable is discrete, yet it is reparametrizable as [Z>p] with Z following standard logistic distribution, whose density and cdf is smooth.
Because of the above many discussions about discrete vs. continuous variables are missleading.
Section 2. The notation of the true distribution as “q” the model as p and the approximate posterior of the model as “q” again is inconsistent. I find the background on ELBO and GANs unnecessary occluding the clarity at this point. For the purpose of introduction, it might be better to give examples of expectation objectives such as:
- dropout: q is the distribution of NN outputs given the input image and integrating out latent dropout noises, gamma are parameters of this NN.
- VAE, GAN: q is the generative model defined as a mapping of a standard multivariate normal distribution by a NN.
- sigmoid belief networks: q is a Bayesian network where each conditional distribution is a logistic regression model.
Then to state to which of these cases the results of the paper are applicable, allow for an improvement of the variance and at what additional computational cost (considering the cost of evaluating the discrete derivatives).
Section 3.
Contrary to the discussion, there are examples of non-negative distributions to which the reparameterization trick can be applied, including log-Normal and Gamma distributions.
Method:
In the case when Rep trick is applicable, is it identical to GO? The difference seems to be only in that the mapping tau may be different from Q^-1. However, this only affects the method of drawing the samples from a fixed known distribution and should have no more effect on the results than say a choice of a pseudo-random number generator. Yet, in Fig.1 some difference is observed between the methods, why is that so?
Sec 7.1
“We adopt the sticking approach hereafter”. Does it mean it is applied with all experiments with GO?
* Related Work
The state of the art allows combining differentiable and non-differentiable pieces of computation:
[Schulman, J., Heess, N., Weber, T., Abbeel, P.: Gradient estimation using stochastic computation graphs.]
I believe it should be discussed in related work. Limitations / where the proposed method brings an improvement should be highlighted.
* Technical Correctness
Equations (5) and (6) require a theorem of differentiating under integral (expectation), such as Leibnitz rule, which in case of (6) requires q_gamma(y)f(y) to be continuous in y and q_gamma(y) continuously differentiable in gamma.
Equation (7) (integration by parts) holds only with some additional requires on f.
Theorem 1 does not take account for the above conditions.


Overfitting is an ubiquitous problem in neural network training and usually mitigated using a holdout data set. Here we challenge this rationale and investigate criteria for overfitting without using a holdout data set. Specifically, we train a model for a fixed number of epochs multiple times with varying fractions of randomized labels and for a range of regularization strengths. A properly trained model should not be able to attain an accuracy greater than the fraction of properly labeled data points. Otherwise the model overfits. We introduce two criteria for detecting overfitting and one to detect underfitting. We analyze early stopping, the regularization factor, and network depth. In safety critical applications we are interested in models and parameter settings which perform well and are not likely to overfit. The methods of this paper allow characterizing and identifying such models.

Overview:
The authors aim at finding and investigating criteria that allow to determine whether a deep (convolutional) model overfits the training data without using a hold-out data set.
Instead of using a hold-out set they propose to randomly flip the labels of certain amounts of training data and inspect the corresponding 'accuracy vs. randomization‘ curves. They propose three potential criteria based on the curves for determining when a model overfits and use those to determine the smallest l1-regularization parameter value that does not overfit.
I have several issues with this work. Foremost, the presented criteria are actually not real criteria (expect maybe C1) but rather general guidelines to visually inspect 'accuracy over randomization‘ curves. The criteria remain very vague and seem be to applicable mainly to the evaluated data set (e.g. what defines a ’steep decrease’?). Because of that, the experimental evaluation remains vague as well, as the criteria are tested on one data set by visual inspection. Additionally, only one type of regularization was assumed, namely l1-regularization, though other types are arguably more common in the deep (convolutional) learning literature.
Overall, I think this paper is not fit for publication, because the contributions of the paper seem very vague and are neither thoroughly defined nor tested.
Detailed remarks:
General:
A proper definition or at least a somewhat better notion of overfitting would have benefitted the paper. In the current version, you seem to define overfitting on-the-fly while defining your criteria.
You mention complexity of data and model several times in the paper but never define what you mean by that.
Detailed:
Page 3, last paragraph: Why did you not use bias terms in your model?
Page 4, Assumption.
- What do you mean by the data being independent? Independent and identically distributed?
- "As in that case correlation in the data can be destroyed by the introduction of randomness making the data easier to learn.“ What do you mean by "easier to learn"? Better generalization? Better training error?
- I don’t understand the assumptions. You state that the regularization parameter should decrease complexity of the model. Is that an assumption? And how do you use that later?
- What does "similar scale“ mean?
Page 4, Monotony.
- You state two assumptions or claims, 'the accuracy curve is strictly monotonically decreasing for increasing randomness‘ and 'we also expect that accuracy drops if the regularization of the model is increased’, and then state that 'This shows that the accuracy is strictly monotonically decreasing as a function of randomness and regularization.‘ Although you didn’t show anything but only state assumptions or claims (which may be reasonable but are not backed up here).
I actually don’t understand the purpose of this paragraph.
- Section 3.3 is confusing to me. What you actually do here is you present 3 different general criteria that could potentially detect overfitting on label-randomized  training sets. But you state it as if those measures are actually correct, which you didn’t show yet.
My main concern here, besides the motivations that I did not fully understand (s.b.), is the lack of measurable criteria. While for criterion 1 you define overfitting as 'above the diagonal line‘ and underfitting as ‚below the line‘, which is at least measurable depending on sample density of the randomization, such criteria are missing for C2 and C3.       Instead, you present vague of ’sharp drops’ and two modes but do not present rigorous definitions. You present a number for C2 in Section 5, but that is only applicable to the present data set (i.e. assuming that training accuracy is 1).
Criterion 2 (b) is not clear.
- I neither understand "As the accuracy curve is also monotone decreasing with increasing regularization we will also detect the convexity by a steep drop in accuracy as depicted by the marked point in the Figure 1(b)"
nor do I understand "accuracy over regularization curve (plotted in log-log space) is constant"?
Does that mean that you assume that whenever the training accuracy drops lower than that of the model without regularization, it starts to underfit?
Due to the lack of numerical measures, the experimental evaluation necessarily remains vague by showing some graphs that show that all criteria are roughly met by regularization parameter \lambda=0.00011 on the cifar data set.  In my view, this evaluation of the (vague) criteria is not fit for showing their possible merit.


ROTDCF: DECOMPOSITION OF CONVOLUTIONAL FIL- TERS FOR ROTATION-EQUIVARIANT DEEP NETWORKS
Explicit encoding of group actions in deep features makes it possible for convolutional neural networks (CNNs) to handle global deformations of images, which is critical to success in many vision tasks. This paper proposes to decompose the convolutional filters over joint steerable bases across the space and the group geometry simultaneously, namely a rotation-equivariant CNN with decomposed convolutional filters (RotDCF). This decomposition facilitates computing the joint convolution, which is proved to be necessary for the group equivariance. It significantly reduces the model size and computational complexity while preserving performance, and truncation of the bases expansion serves implicitly to regularize the filters. On datasets involving in-plane and out-of-plane object rotations, Rot-DCF deep features demonstrate greater robustness and interpretability than regular CNNs. The stability of the equivariant representation to input variations is also proved theoretically. The RotDCF framework can be extended to groups other than rotations, providing a general approach which achieves both group equivariance and representation stability at a reduced model size.

This work extends on [1] by constructing CNN filters using Fourier-Bessel (FB) bases for rotation equivariant networks. Additionally to [1] it extends the process with using SO(2) bases which allow to learn combination of rotated FB bases and ultimately achieve good performance with less parameters than standard CNN networks thanks to filter truncation.
In general, this work is well written and shows interesting results. However it lacks context with regards to other existing works. For example [2] also uses steerable filters for achieving rotation equivariance, however with different steerable bases (rotation harmonics instead of FB). It would be useful to clarify why FB bases are more appropriate for truncation, eventually providing empirical evidence (even though rotation harmonics would probably need more parameters). Authors mention [2], however disregard it due to computational complexity, which would be the same if the rotation harmonics bases were truncated as well.
Similarly, this work is not strong in evaluating against existing methods. It provides evaluation of the vanilla group equivariant networks in a similar configuration, but due to design choices in the training and test set, it is not possible to compare it against other algorithms and other steerable bases such as those from [2]. This degrades the results slightly as it does not allow to verify the baseline results from other works.
Additionally, it would be useful to provide an ablation study which would show how important the bases in SO(2) are important for the model accuracy. This would allow to compare the results against the [1] as the FB filters are steerable as well (Equation 4).
It is hard to reach a final rating for this submission. On one hand, it can be seen as an incremental improvement of [1] for a new domain of tasks, without a thorough comparison against existing methods. On the other hand, the paper is well written and the results look promising - evaluation verifies that the algorithm performs well in multiple tasks with a fraction of parameters.
Considering that authors plan to release the source code and that this conference aims for publishing novel ideas (and the goal of this work is to achieve rotation equivariance with less parameters, which hasn't been tackled before), I am inclined towards acceptance of this paper, even though the experiments can be significantly improved.
Unfortunately, I was not able to verify correctness of the provided proofs.
Additional minor issues:
* The paper does not specify what FB bases exactly are being used (such as in [table 1;1]), mainly it does not seem to specify the SO(2) bases.
* It would be useful to visualise K and K_\alpha in Figure 1.
* Citations, if not part of the sentence, should be in parentheses to improve readability (\citep for natbib).
* On page 8, end of first paragraph - wrong reference (see S.M.)
* L, in section 2.3 is not defined.
[1] Qiu, Qiang, et al. "DCFNet: Deep Neural Network with Decomposed Convolutional Filters.", ICML 2018
[2] Weiler, Maurice, et al. “Learning Steerable Filters for Rotation Equivariant CNNs.” CVPR 2018


We propose an approach to generate realistic and high-fidelity stock market data based on generative adversarial networks. We model the order stream as a stochastic process with finite history dependence, and employ a conditional Wasserstein GAN to capture history dependence of orders in a stock market. We test our approach with actual market and synthetic data on a number of different statistics, and find the generated data to be close to real data.

This paper proposes a Generative Adversarial Network (GAN) methodology to learn the distribution of limit orders that arrive on an equity market. The proposed approach captures the (mostly discrete) structure in the generated orders; modeling is carried out with a conditional Wasserstein-GAN with a recurrent neural networks in both the generator and critic used to capture the time dependency, and convolutional layers to capture other conditioning information. Experiments are provided on both synthetic and real data.
Overall, the paper is well written and easy to follow. The application of WGAN to modeling financial market microstructure is novel and appropriate: microstructure is probably one of the areas of finance where generative models can be trained with ample data, and overfitting risks can therefore be controlled to some extent. In this respect, the paper brings a valuable contribution, both to finance by proposing a direct generative model of a complex process, and to machine learning by showing a useful application of GANs outside of the traditional domains.
My main reservation with the paper is that the experimental results could be more convincing. In particular, results for real data are shown for only two stocks, representing respectively 20k and 230k orders. Although the trained model captures some aspects of the distribution (e.g. inter-arrival times), it misses others, such as important aspects of the (univariate) price distribution, as well as the best bid-ask dynamics. As they stand, the results appear mostly anecdotal, and lack either a breadth or depth of analysis. It would be sensible for the authors to:
1. Compare their models to other limit-order book simulation models proposed in the financial econometrics literature;
2. Characterize the sensitivity of the learned distribution on the architecture hyperparameters.
For these reasons, it appears difficult to recommend acceptance of the paper in its current state.


Adversarial attacks on neural networks perturb the input at test time in order to fool trained and deployed neural network models. Most attacks such as gradientbased Fast Gradient Sign Method (FGSM) by Goodfellow et al. (2015) and DeepFool by Moosavi-Dezfooli et al. (2016) are input-dependent, small, pixel-wise perturbations, and they give different attack directions for different inputs. On the other hand, universal adversarial attacks are input-agnostic and the same attack works for most inputs. Translation or rotation-equivariant neural network models provide one approach to prevent universal attacks based on simple geometric transformations. In this paper, we observe an interesting spectral property shared by all of the above input-dependent, pixel-wise adversarial attacks on translation and rotation-equivariant networks. We exploit this property to get a single universal attack direction that fools the model on most inputs. Moreover, we show how to compute this universal attack direction using principal components of the existing input-dependent attacks on a very small sample of test inputs. We complement our empirical results by a theoretical justification, using matrix concentration inequalities and spectral perturbation bounds. We also empirically observe that the top few principal adversarial attack directions are nearly orthogonal to the top few principal invariant directions.

This paper studies the problem of computing non-data specific perturbations, also known as universal perturbations, to attack neural networks and take profit of their inherent vulnerability. Compared to previous works in the domain, the authors look specifically at equivariant networks, and derive geometric insights and methods to compute universal perturbations for these networks.
The paper starts by analysing the main/principal directions of set of perturbations that are able to change the decisions in different forms of equivariant neural networks. With this heuristic study, a few main directions are shown to be shared by most adversarial perturbations. The authors then propose to construct universal perturbations built on the insights given by the principle directions of perturbations, which is an interesting an effective method. In addition, it is shown that a few adversarial samples are sufficient to identify pretty  accurately the principle directions. The fooling rates achieved by this method is pretty good, which demonstrates that the proposed strategy is reasonable.
The key idea in this paper (using principal shared directions of perturbations, computed on a small subset of data points) has unfortunately already been proposed and tested in classical (non-equivariant) neural networks - see for example Fig 9 in Moosavi-Dezfooli, 2017, cited in the paper, and published in CVPR 2017. The present paper proposes however a few additional bits of information with a nice theoretical analysis, while the previous works were mostly based on heuristics. It is probably not sufficient however to pass the cut in ICLR.
The interesting additional novelty here is the study of equivariant networks. However, this ends up falling sort of initial expectations - there seems to be nothing specific to equivariant networks in the proposed study, and the solution and algorithm is actually applicable to any neural network architectures (?). Also, no specific insights are derived for equivariant networks, which could be potentially very interesting to make progress in understanding better equivariant representations, which still consist in a widely open research problem.
In general, the paper has a non-classical organisation, with a lot of heuristics that are not discussed in depth - that gives a sort of high-level impression that the proposed idea is potentially nice, but that but superficially addressed. It should probably be improved in the next versions of this work.


Imitation learning aims to learn an optimal policy from expert demonstrations and its recent combination with deep learning has shown impressive performance. However, collecting a large number of expert demonstrations for deep learning is time-consuming and requires much expert effort. In this paper, we propose a method to improve generative adversarial imitation learning by using additional information from non-expert demonstrations which are easier to obtain. The key idea of our method is to perform multiclass classification to learn discriminator functions where non-expert demonstrations are regarded as being drawn from an extra class. Experiments in continuous control tasks demonstrate that our method learns better policies than the generative adversarial imitation learning baseline when the number of expert demonstrations is small.

The paper proposes a modification of GAIL (Ho & Ermon, 2016) to make use of non-expert data. The non-expert data is used by training a classifier to classify between roll-outs of the current policy, expert demonstrations and non-expert demonstrations. Similar to GAIL, the policy is iteratively updated using TRPO with a cost that is given by the log probability of predicting the policy. The use of non-expert data acts as regularization in order to learn better features similar to universum prescription (Zhang & LeCun, 2017).
The paper is well-written and very clear. The general problem setting is interesting, but I think it is of rather little significance, because I do not see many clear applications. The evaluation focuses on simulated robots, however gathering non-expert data on real robots would be very expensive, so the approach would not make a lot of sense here (even if we replace TRPO a more sample efficient rl method). The paper mentions the game of Go, but learning a policy on such large state spaces is not feasible without major modification and significant computational effort. However, the paper also mentions autonomous driving, which might be a more convincing application, because we can have a lot of demonstrations that we do not want to label as expert trajectories. I think the paper would profit a lot from having an experiment where the importance of making use of non-expert data becomes evident.
The approach seems sound, although I think that we can not expect much benefit from using the unlabelled data in the proposed way. By not making any assumptions on the non-expert data, they do not carry any information about the objective; the information that they carry about the system dynamics is not exploited for the RL update. Instead, the use of non-expert data is restricted to learning better features for discriminating between the agent and the expert. However, non-expert data is typically not cheaper than policy roll-outs and better features could also be learned by using more samples from the policy. The experiments also show only slight benefits, especially when comparing the final performance (instead of total returns) and accounting for the additional system interactions needed for generating the non-expert data. To make the comparison fairer, we could consider using a few more system interactions (variable K in the paper) per iteration for standard GAIL, so that the total number of function evaluations would match those of M-GAIL after a certain number of iterations. Especially if K is appropriately tuned, it is not clear whether we could still show an advantage of M-GAIL. It would also be interesting to show, whether we can benefit from using the policy roll-outs of previous iterations as non-expert data for the current iteration (in the traditional IL setting where no non-expert data is available a priori).
The main weakness of the paper is, that the novelty seems marginal. Instead of doing binary classification with cross-entropy loss, we're doing three-class classification with cross-entropy loss and use it for binary classification (by throwing away the auxiliary logit for predicting the non-expert class). Did I miss any other difference to GAIL? We can argue whether the policy objective is different (to me, H_\phi of M-GAIL corresponds to the discriminator of GAIL and the objective is exactly the same), however, even if we call it a minor modification, we would have very little novelty in the approach. As the paper does also not compensate for this with very good results or thorough theoretical analysis, I think that the contribution is too minor.
I do not see the point of section 4.4. and the related appendix A2. For all I understand, it proves that when using lambda=0 (standard GAIL, right?), the proof of Fu et al. (2018) for GAIL is valid (i.e. we learn a completely useless reward function that does not carry any additional information compared to the policy), and when using lambda!=0 we learn something different. I don't see the the purpose of this statement and I don't think that it needs to be proven. The paper argues, that for small lambda we can treat the discriminator logits as approximations of these (completely useless) reward functions--without providing any bound. As I do not see why this would be useful, I think the section should be removed.
Minor:
Typo: "[...]due to its dependent[sic] on the linearity of reward functions and good feature engineering"


Learning to control an environment without hand-crafted rewards or expert data remains challenging and is at the frontier of reinforcement learning research. We present an unsupervised learning algorithm to train agents to achieve perceptuallyspecified goals using only a stream of observations and actions. Our agent simultaneously learns a goal-conditioned policy and a goal achievement reward function that measures how similar a state is to the goal state. This dual optimization leads to a co-operative game, giving rise to a learned reward function that reflects similarity in controllable aspects of the environment instead of distance in the space of observations. We demonstrate the efficacy of our agent to learn, in an unsupervised manner, to reach a diverse set of goals on three domains -Atari, the DeepMind Control Suite and DeepMind Lab.

Summary:
The authors take up an important problem in unsupervised deep reinforcement learning which is to learn perceptual reward functions for goal-conditioned policies without extrinsic rewards from the environment. The problem is important in order to push the field forward to learning representations of the environment without predicting value functions from scalar rewards and learn more generalizable aspects of the environment (the authors call this mastery) as opposed to just memorizing the best sequence of actions in typical value/policy networks.
Model-based methods are currently hard to execute as far as mastery is concerned and goal-conditioned value functions are a good alternative. The authors, therefore, propose to learn UVFA (Schaul et al) with a learned perceptual reward function r(s, s_g) where 's' and 's_g' are current and goal observations respectively. They investigate a few choices for deriving this reward, such as pixel-space L2 distance, Auto-Encoder feature space, WGAN Discriminator (as done in SPIRAL - Ganin and Kulkarni et al), and their approach: cosine similarity based log-likelihood for similarity metric (as in Matching Networks).  They show that their approach works better than other alternatives on a number of visual goal-based tasks.
Specific aspects:
1. A slight negative: I find the whole pipeline extremely hacky and raises serious questions on whether this paper/technique is easy to apply on a wide variety of tasks. It gives me the suspicion that the environments were cherry-picked for showing the success of the proposed method, though, that's, in general, true of most deep RL papers. It would be nice if the authors instead wrote the paper from the perspective of proposing a new benchmark (it would be amazing if the benchmark is open sourced so that it will lead to more people working specifically on this setting and a lot more comparisons).
-- Revision: The pipeline is hacky, but getting GAN based reward learning to work is also not very straightforward. The authors do plan to release the detectors used for the benchmarking.
2. To elaborate on the above, these are the portions I find hacky:
(i) Need for decoy observations to learn an approximate log-likelihood
(ii) Using sparse reward for all transitions except the final terminal state: Yes, I am aware of the fact that HER has already shown sparse rewards are easier to learn value functions with, compared to dense rewards. But I am genuinely surprised that you have pretty much the same setting (ie re-label only terminal transition, r(s_T, s_g)) and motivate the need for learning a perceptual metric. If the information bits per transition is similar to HER in terms of the policy network's objective function, I am not sure why you need to learn a perceptual reward then? There's also no baseline comparison with just naive HER on image observations. That will be worth seeing actually. I feel this kind of comparisons are more interesting and important for the message of the paper. Note that in other papers cited in this, such as SPIRAL, UPN, etc, the reward metrics are used for every state transition.
(iii) In addition to naive image HER, I would really like to see a SPIRAL + HER baseline as is. ie use the GAN reward for all transitions and also use relabeling for successes. My prior belief is that this will work really well. I would really like to know how the reward for each transition in the trajectory works (both for SPIRAL and your approach) and how the naive HER works.
--Revision: The authors have added HER baselines. Agreed with the authors that comparison of per-timestep perceptual reward vs terminal state perceptual reward is a good topic for future work.
3. Another place I really found confusing throughout the paper is the careless swapping of notations, especially in the xi(h) and e(h). Please use consistent notations especially in equation (3), the pseudocode and the rest of the paper.
4.  a. Would be nice to know if a VAE feature space metric is bad, but not a strict requirement if you don't have time to do it. But I think showing Euclidean metric baseline on VAE is better than an AE.
      b. Another baseline that is related is to learn a metric with a triplet loss as in Sermanet's work. Or any noise contrastive loss approach (such as CPC). The matching networks approach is similar in spirit. Just pointing out as reference and something worth trying, but not expecting it to be done for rebuttal.

5. Overall, I think this is a good paper, gives a good overview of an important problem; the matching networks idea is nice and simple; but the paper could be more broader in terms of writing than trying to portray the success of DISCERN specifically. I would be happy accepting it even if the SPIRAL baseline or VAE / AE baseline worked as well as the matching networks because I think those approaches are more principled and likely to require fewer hacks and could be applied to a lot of domains easily. I also hope the authors run the baselines I asked for just to make the paper more scientifically complete.
6. Not a big deal for me in terms of deciding acceptance, but for the sake of good principles in academics, related work could be stronger, though I can understand it must have been small purely due to page limits.
Some papers which could be cited are (1) Unsupervised Perceptual Rewards (though it uses AlexNet pre-trained), (2) Time Contrastive Networks (which also uses AlexNet and doesn't really work on single-view tasks but is a good citation to add), (3) Original UVFA  (definitely has to be there given you even use the abbreviation for the keywords description of the paper)
7. Some slightly incorrect facts/wording in the paper: The two papers cited in model-based methods (Oh and Chiappa) are not really unsupervised. They use a ton of demonstrations to learn those world models. Better citation might be David Ha's World Models or Chelsea Finn's Video Prediction.


HYPERBOLIC ATTENTION NETWORKS
Recent approaches have successfully demonstrated the benefits of learning the parameters of shallow networks in hyperbolic space. We extend this line of work by imposing hyperbolic geometry on the embeddings used to compute the ubiquitous attention mechanisms for different neural networks architectures. By only changing the geometry of embedding of object representations, we can use the embedding space more efficiently without increasing the number of parameters of the model. Mainly as the number of objects grows exponentially for any semantic distance from the query, hyperbolic geometry -as opposed to Euclidean geometry-can encode those objects without having any interference. Our method shows improvements in generalization on neural machine translation on WMT'14 (English to German), learning on graphs (both on synthetic and real-world graph tasks) and visual question answering (CLEVR) tasks while keeping the neural representations compact.

This paper replaces the dot-product similarity used in attention mechanisms with the negative hyperbolic distance, and applies this new attention to the existing Transformer model, graph attention networks (GAT), and Relation Networks (RN). Accordingly, they use Einstein midpoint to compute the aggregation weights of attention in hyperbolic space. The idea of using hyperbolic rather than Euclidean space is based on the assumption that the input embeddings (neural net activations) are on the hyperbolic manifold, which follows power-law distribution and can be seem as a smooth description of tree-like hierarchy of data points. This assumption might hold for small neural networks with relatively low dimensional output. One main reason why this paper adopts the hyperbolic space is that the volume of hyperbolic space grows exponentially with the increase of radius while that of Euclidean space grows only polynomially. Using hyperbolic distance can increase the capacity of networks and handle the complexity of data. Experiments on Transformer and relation network show that Transformer, GAT and RN with the new attention metric produce better performance than Euclidean distance.
Pros:
1. Comparing to the existing methods using representations for shallow models in hyperbolic geometry, this paper extends the idea to deep neural networks.
2. The proposed attention mechanism can be easily applied to many of existing networks to enhance their capacity.
3. The experiments show several interesting results: 1) hyperbolic recursive transformer (RT) is consistently superior to Euclidean RT across the tasks in this paper; 2) hyperbolic space substantially benefits the low-capacity networks (i.e., low-dimensionality hidden state); 3) Einstein midpoint is better than Euclidean aggregation in hyperbolic space; 4) using sigmoid rather than softmax to compute attention weight may achieve better effectiveness on some tasks for the reason that the attention weights over different entities ay do not compete with each other.
Cons:
1. The novelty of this paper is replacing the Euclidean metric with another existing metric, which has already been used in previous ML models. So the contribution is limited.
2. As explicitly claimed in the paper and also reflected by the experimental results (e.g., Transformer-Big in Table 2). The hyperbolic metric only brings noticeable improvement to small neural nets with limited compacity on relatively small datasets. When applied it to most SOTA models (which are usually large/deep/wide neural networks) on larger datasets, it loses the advantage. This fact might seriously limit the application of the proposed technique.
3. Small models are preferred for inference especially on edge devices. But model compression and knowledge distillation can make small models having similar performance as large models, which might be much better than directly training a small model with the proposed metric.
4. Although hyperbolic metric reflects the power-law distribution, which is a very natural assumption verified on many kinds of real data (social networks and physical statistics), I am not fully convinced that it still holds on an embedding space produced by a neural net (since attention are usually applied to the outputs of a neural net).
5. In the experiments, does the model with the proposed metric cost similar training/inference time comparing to the baselines? What is the trade-off between improvements and extra time costs? I notice that the results of the proposed attention in Table 2 are ~0.5% higher than the results from the earlier arXiv version of this paper. What is the reason for the improvements? Did you increase the training steps?


In recent years, various studies have focused on the robustness of neural nets. While it is known that neural nets are not robust to examples with adversarially chosen perturbations as a result of linear operations on the input data, we show in this paper there could be a convex polytope within which all examples are misclassified by neural nets due to the properties of ReLU activation functions. We propose a way to find such polytopes empirically and demonstrate that such polytopes exist in practice. Furthermore, we show that such polytopes exist even after constraining the examples to be a composition of image patches, resulting in perceptibly different examples at different locations in the polytope that are all misclassified.

This paper presents an algorithm for finding a polytope of adversarial examples. This means that within a convex hull, you can move around freely and get a new adversarial example at each point, while still maintaining misclassification. It then couples this with a method of generating nearest neighbor patch-based images in an effort to create "macro-level different" examples. The premise is interesting, but the implications are questionable and I do not find the work in macro-level differences to be sound. This could be based in misunderstandings, so please let me know if you think that is the case.
Strengths:
- The notion of the polytope is interesting and the algorithm for finding such polytope seems perfectly reasonable.
- I think the goal of macro-level adversarial examples is interesting.
Weaknesses:
- First of all, the 5 corners of the polytope all look the same to me (for instance fig 2). This is not encouraging, because it means that every single point in the polytope will also look exactly like the corners. To be frank, this means the polytope is not that interesting and has only found an extremely small pocket of adversarial examples. If you use a regular method of finding a single adversarial example, I'm sure the outcome wouldn't change within some ball around the sample (perhaps with very small radius, but nonetheless). In fact, a comparison between that ball's volume and the volume of the polytope would be interesting.
- The implication of these polytopes is not at all clear if it doesn't really allow us to generate adversarial example of a new flavor. The investigation into macro-level differences does not help the case, as I will explain.
- I am not at all convinced that there is any meaning to the examples with "macro-level differences."  It's a bit unclear to me how many patches are used per image, but assuming that a patch is centered over each pixel,  it would mean that we have as many control parameters as we have pixels, which assuming the pixels each have three color values, is just 1/3 of the original degrees of freedoms. Now, the patches probably do constrain what we can paint a bit, but since the patches are applied with a pyramid, it means the center pixel will contribute more than any other for a given patch, so I'm not so sure. I'm not convinced that we can't come up with linear combinations of these patches that produce highly non-natural images with "micro-level" adversarial patterns. In fact, I think section 4.1 and figure 7 provide evidence to the contrary. Let me explain:
    - Section 4.1: Why do you need a total variation penalty at all if you have constructed a patch-based drawing method that is supposed to be unable to produce unnatural high-frequency patterns? If you only had a handful of patches and they were all non-overlapping, then this would be impressive and.
    - Figure 7: We can clearly see high-frequency patterns that create the shadow of an obelisk in 7(a). I think the same is true for "erase", although the pattern is not as recognizable. The examples actually look more suspicious than regular adversarial examples, since it looks like the original image has simply been blurred, which means the adversarial perturbations are more clear. I understand that these patterns were created using a complicated scheme of natural patches, but I think you made this method too powerful. The one interesting quality is the bottom right of the trimaran which looks like a shark - however, that is a singular occurrence in your examples and it certainly feels like the high-frequency patterns will contribute much more to class than the shark itself.
- Please let me know if I am misinterpreting the importance of the results in Figure 7, since this is an important culmination of this work.
Other comments:
- Some of notation is a bit confusing. In (1), why is p not bold but x and t are bold? They are all vectors. In Algorithm 1, x is not bold anymore.
- Algorithm 1 also seems quite unnecessary to include so explicitly.
- Isn't a bounded polytope called a "simplex"? Perhaps there is a distinction that I'm not aware of, but the absence of the word "simplex" throughout the whole paper surprised me a bit. Perhaps this is a perfectly correct omission due to differences that I'm not aware of.
Minor comments:
- abstract, "We propose a way to finding" -> either "to->"of" or "find"
- page 3, "and we can generate new colliding example" -> "a new colliding example"
- page 3, "taking arbitrary an convex combinations" -> "combination"
- page 3, "Given a target x", I think you mean "Given a target t"
- page 5, "As many gradient-based method" -> "methods"
- page 8, "carton"? "rubber"? Those are not in figure 7(b).
- page 10, "are crucial to less non-robust" ? This sentence (which is the final sentence of the conclusion and thus has a certain level of importance) is not something that is novel to your paper. The impact of non-linearities on adversarial examples have been well-studied.


Proximal Policy Optimization (PPO) is a highly popular model-free reinforcement learning (RL) approach. However, in continuous state and actions spaces and a Gaussian policy -common in computer animation and robotics -PPO is prone to getting stuck in local optima. In this paper, we observe a tendency of PPO to prematurely shrink the exploration variance, which naturally leads to slow progress. Motivated by this, we borrow ideas from CMA-ES, a black-box optimization method designed for intelligent adaptive Gaussian exploration, to derive PPO-CMA, a novel proximal policy optimization approach that can expand the exploration variance on objective function slopes and shrink the variance when close to the optimum. This is implemented by using separate neural networks for policy mean and variance and training the mean and variance in separate passes. Our experiments demonstrate a clear improvement over vanilla PPO in many difficult OpenAI Gym MuJoCo tasks.

This paper proposes an improvement of the PPO algorithm inspired by some components of the CMA-ES black-box optimization method. The authors evaluate the proposed method on a few Mujoco domains and compare it with PPO method using simpler exploration strategies. The results show that PPO-CMA less likely to getting stuck in local optima especially in Humanoid and Swimmer environments.
Major comments:
The reason that CMAES discards the worst batch of the solutions is that it cannot utilize the quality of the solutions, i.e., it treats every solution equally. But PPO/TRPO can surely be aware of the value of the advantage, and thus can learn to move away from the bad area. The motivation of remove the bad samples is thus not sound, as the model cannot be aware of the areas of the bad samples and can try to repetitively explore the bad area.
Please be aware that CMAES can get stuck in local optima as well. There is no general convergence guarantee of CMAES.
Detailed comments:
- In page 4, it is claimed that "actions with a negative $A^\pi$ may cause instability, especially when one considers training for several epochs at each iteration using the same data" and demonstrate this with Figure 2. This is not rigorous. If you just reduce all the negative advantage value to zero and calculate its gradient, the method is similar to just use half of step-size in policy gradient. I speculate that if you halve the step-size in "Policy Gradient" setting, the results will be similar to the "Policy Gradient(only positive advantages)" setting. Furthermore, different from importance sampling technique, pruning all the negative advantage will lose much **useful** information to improve policy. So I think this is maybe not a perfect way to avoid instability although it works in experiments.
- There have been a variety of techniques proposed to improve exploration based on derivative-free optimization method. But in my opinion, the way you combine with CMA-ES to improve exploration ability is not so reasonable. Except for the advantage function is change when policy is updated (which has mentioned in "D LIMITATIONS"), I consider that you do not make good use of the exploration feature in CMA-ES. The main reason that CMA-ES can explore better come from the randomness of parameter generation (line 2 in Algorithm 2). So it can generate more diverse policy than derivative-based approach. However, in PPO-CMA, you just replace it with the sampling of policy actions, which is not significant benefit to exploration. It more suitable to say that you "design a novel way to optimize Gaussian policy with separate network for it mean and variance inspired by the CMA-ES method rather than "provides a new link between RL and ES approaches to policy optimization" (in page 10).
- In experiments, there are still something not so clear:
1. In Figure 5, I notice that the PPO algorithm you implemented improved and then drop down quickly in Humanoid-v2 and InvertedDoublePendulum-v2, which like due to too large step-size. Have you tried to reduce it? Or there are some other reasons leading to this phenomenon.
2. What's the purpose of larger budget? You choose a bigger iteration budget than origin PPO implementation.
3. What the experiments have observed may not due to the clipping of negative reward, but could due to the scaling down of the reward. Please try reward normalization.


Models optimized for accuracy on single images are often prohibitively slow to run on each frame in a video. Recent work exploits the use of optical flow to warp image features forward from select keyframes, as a means to conserve computation on video. This approach, however, achieves only limited speedup, even when optimized, due to the accuracy degradation introduced by repeated forward warping, and the inference cost of optical flow estimation. To address these problems, we propose a new scheme that propagates features using the block motion vectors (BMV) present in compressed video (e.g. H.264 codecs), instead of optical flow, and bi-directionally warps and fuses features from enclosing keyframes to capture scene context on each video frame. Our technique, interpolation-BMV, enables us to accurately estimate the features of intermediate frames, while keeping inference costs low. We evaluate our system on the CamVid and Cityscapes datasets, comparing to both a strong single-frame baseline and related work. We find that we are able to substantially accelerate segmentation on video, achieving near real-time frame rates (20+ frames per second) on large images (e.g. 960×720 pixels), while maintaining competitive accuracy. This represents an improvement of almost 6× over the single-frame baseline and 2.5× over the fastest prior work.

# Paper summary
This paper advances a method for accelerating semantic segmentation on video content at higher resolutions. Semantic segmentation is typically performed over single images, while there is un-used redundancy between neighbouring frames. The authors propose exploiting this redundancy and leverage block motion vectors from MPEG H.264 video codec which encodes residual content between keyframes. The block motion vectors from H264 are here used to propagate feature maps from keyframes to neighbouring non-keyframe frames (in both temporal directions) avoiding thus an additional full forward pass through the network and integrate this in the training pipeline. Experimental results on CamVid and Cityscapes show that the proposed method gets competitive results while saving computational time.
# Paper strengths
- This paper addresses a problem of interest for both academic and industrial purposes.
- The paper is clearly written and the authors argument well their contributions, adding relevant plots and qualitative results where necessary.
- The two-way interpolation with block motion vectors and the fusion of interpolated features are novel and seem effective.
- The experimental results, in particular for the two-way BMV interpolation, are encouraging.
# Paper weaknesses
- The idea of using Block Motion Vectors from compressed videos (x264, xvid) to capture motion with low-cost has been previously proposed and studied by Kantorov and Laptev [i] in the context of human action recognition. Flow vectors are obtained with bilinear interpolation from motion blocks between neighbouring frames. Vectors are then encoded in Fisher vectors and not used with CNNs as done in this paper. In both works, block motion vectors are used as low-cost alternatives to dense optical flow. I would suggest to cite this work and discuss similarities and differences.
- Regarding the evaluation of the method, some recent methods dealing with video semantic segmentation, also using ResNet101 as backbone, are missing, e.g. low latency video semantic segmentation[ii]. Pioneer Clockwork convnets are also a worthy baseline in particular in terms of computational time (results and running times on CityScapes are shown in [ii]). It would be useful to include and compare against them.
- In Section 4.1.2 page 7 the authors mention a few recent single-frame models ((Yu et al. (2017); Chen et al. (2017); Lin et al. (2017); Bilinski & Prisacariu (2018)) as SOTA methods and the current method is competitive with them. However I do not see the results from the mentioned papers in the referenced Figures. Is this intended?
- On a more general note related to this family of approaches, I feel that their evaluation is usually not fully eloquent. Authors compare against similar pipelines for static processing and show gains in terms of computation time. The backbone architecture, ResNet-101 is already costly for high-resolution inputs to begin with and avoiding a full-forward pass brings quite some gains (though a part of this gain is subsequently attenuated by the latency caused by the batch processing of the videos). There are recent works in semantic segmentation that focus on architectures with less FLOPs or memory requirements than ResNet101, e.g. Dilated ResNets [iii], LinkNet[iv]. So it could be expected that image-based pipelines to be getting similar or better performance in less time. I expect the computational gain on such architectures when using the proposed video processing method to be lower than for ResNet101, and it would make the decision of switching to video processing or staying with frame-based predictions more complex.
The advantage of static image processing is simpler processing pipelines at test time without extra parameters to tune. It would be interesting and useful to compare with such approaches on more even grounds.
# Conclusion
This paper takes on an interesting problem and achieves interesting results. The use of Block Motion Vectors has been proposed before in [i] and the main novelty of the paper remains only the interpolation of feature maps using BMVC. The experimental section is missing some recent related methods to benchmark against.
This work has several strong and weak points. I'm currently on the fence regarding my decision. For now I'm rating this work between Weak Reject and Borderline
# References
[i] V. Kantorov and I. Laptev, Efficient feature extraction, aggregation and classification for action recognition, CVPR 2014
[ii] Y. Li et al., Low-Latency Video Semantic Segmentation, CVPR 2018
[iii] F. Yu et al., Dilated Residual Networks, CVPR 2017
[iv] A. Chaurasia and E. Culurciello, LinkNet: Exploiting Encoder Representations for Efficient Semantic Segmentation, arXiv 2017


Existing neural networks are vulnerable to "adversarial examples"-created by adding maliciously designed small perturbations in inputs to induce a misclassification by the networks. The most investigated defense strategy is adversarial training which augments training data with adversarial examples. However, applying single-step adversaries in adversarial training does not support the robustness of the networks, instead, they will even make the networks to be overfitted. In contrast to the single-step, multi-step training results in the state-of-the-art performance on MNIST and CIFAR10, yet it needs a massive amount of time. Therefore, we propose a method, Stochastic Quantized Activation (SQA) that solves overfitting problems in single-step adversarial training and fastly achieves the robustness comparable to the multi-step. SQA attenuates the adversarial effects by providing random selectivity to activation functions and allows the network to learn robustness with only single-step training. Throughout the experiment, our method demonstrates the state-of-the-art robustness against one of the strongest white-box attacks as PGD training, but with much less computational cost. Finally, we visualize the learning process of the network with SQA to handle strong adversaries, which is different from existing methods.

The paper proposes a model to improve adversarial training, by introducing random perturbations in the activations of one of the hidden layers. Experiments show that robustness to attacks can be improved, but seemingly at a significant cost to accuracy on non-adversarial input.
I have not spent significant time on adversarial training, and review the paper under the following understanding: It was observed that the decision regions of a class are sprinkled with "holes" that get misclassified. These holes are neither naturally occuring. Their existence allows a potential attacker to coerce a model into mis-classifying by providing specially crafted inputs, in order to attain a benefit. Therefore, those holes are called "adversarial" examples. The risk is heightened by the fact that adversarial examples are commonly not mis-classified by humans (or even detectable by the eye). To "plug" the holes, one includes adversarial examples in the training, called "adversarial training." A resulting system should now have a much improved accuracy for the "holes", while ideally not affecting classification accuracy for the natural examples, which will continue to constitute nearly 100% of the samples the system will be used on. (The "hole" metaphor may not be entirely appropriate, since the space of adversarial examples that are neither misclassified by humans nor detectable is likely much larger than the space of naturally occuring samples.)
The paper proposes a way of plugging the hole by quantizing layer activations. The results show that this makes the system robust to adversarial attacks.
Clarity:
I spent a lot of time figuring out, as someone who has not spent a lot of time with this, what is being evaluated. It is very unclear whether the non-clean systems in Tables 1 and 2 do apply FGSM etc. also in training (in combination with SQA), or only to the test samples. Table 4, the wording in 4.2, and the wording of the Conclusion indicate that they are. But then, where do I find the accuracy on the naturally-occuring (non-manipulated) samples?
The only combination of interpretations that makes sense in the end is to parse "The networks are all trained with fast single-step adversaries" as to mean "The networks are all trained with FGSM", and that the non-Clean columns in Table 1 refer to test data perturbed by the respective method, while the Clean column shows the accuracy on the natural data. This *must* be clarified in the final version, as it took way too long to understand this. I strongly suggest to do this with the naming: change small_full to small_FGSM, and small_SQA to small_SQA+FGSM.
Assuming I figured this out right, the tables still lack the baseline accuracy of doing nothing (clean-clean), so one can know how much the nearly-100% use case gets affected.
Results:
The second concern I have is that, assuming my reading of the results as described above is correct, that the SQA method quite severely affects accuracy on the clean test data, e.g. increasing the error rate on CIFAR by 72% (from 12.33% to 17.06%). There must be a discussion on why such severe performance hit is worth it, especially since there often is an accuracy cliff below which there is a steep loss of usability of a system. For example, according to my personal experience in speech recognition, the difference between 12% and 17% is the difference between decent and unacceptable user experience (also considering that a few percent of errors are caused by ambiguities in the ground-truth annotations themselves, which should be the case for CIFAR as well).
Figure 1 seems a little misleading in this regard since the areas of good accuracy are very condensed. It should be rescaled, as only the area close to the optimum performance is relevant. It does not matter whether we degrade from 99.x% to 77% or 58%, or even 95-ish. All of those hurt performance to the point of not being useful.
It would be nice to discuss what an accuracy metric would be that is useful for the end user. It would have to be a combination of the expected cost of a misclassification of a natural image and the expected cost caused by attacks. A good method would improve this overall metric. A paper attempting to address adversarial attacks should at least discuss this topic briefly, in my view.
Technical soundness:
A technical question I have is whether the min-max normalization may be too susceptible to outliers. A single extreme activation can drastically shift the threshold for \lambda=1. How about a mean-var normalization? If there is batch or layer normalization in the system, your activations may already be scaled into a consistent range anyway, that might allow you to use a constant scaling on top of that.
Another question I have is: quantization is often modeled as adding uniform noise. Why not add noise directly? And why uniform noise? For example, would compute g = h + Gaussian noise with std dev=(max-min)/lambda work equally well? What is special about quantization?
And another technical question: My guess is that the notable loss of accuracy is caused by the strong quantization (two values only in the case of \lambda=1). I think the paper should show results for larger lambdas, specifically whether there is a better trade-off point between the accuracy loss from quantization vs. robustness to adversarial samples.
Section 3/SQA: "This is the reason why we rescale g^i to the original range of h^i" This seems wrong. I think the main reason is that one would not want to totally change the dynamic ranges of the network, as it may affect convergence merely by scaling. You'd want to limit any impact on convergence to the quantization itself.
Significance:
I think the significance is limited. Given that the accuracy impact of the mitigation method is very large, I do not consider this paper as substantially solving the problem, or even bringing a practical solution much closer in reach.
Pros:
 - tnteresting idea;
 - comparison against various attacks.
Cons:
 - Hard to understand because it was left unclear what is evaluated, at least to readers who are not familiar with a possibly existing implied convention;
 - The method seems to harm accuracy on clean data a lot, which is the main use case of such a system.
I would in the current form reject the paper. To make it acceptable, the clarity of presentation, especially of the results, must be improved, but more importantly, more work seems necessary to reduce the currently significant accuracy hit from the method, and the trade-off of quantization level vs. robustness should be addressed.
Minor feedback:
Please review the paper for grammar and spelling errors (e.g. "BinaryConnect constraints" or the use of "make", which is often not correct).
In Algorithm 1, I suggest to not use 'g', as it may be mis-read as "gradient." Unless this is a common symbol in this context.
"Thus, we propose SQA" warrants another \subsubsection{}, to indicate where \subsubsection{BinaryConnect} ends.
Section 2.2's early reference to SQA is a little confusing, since SQA has not formally been defined. I would smooth this a little, e.g. change "SQA can be considered" to "We will see that our SQA, as introduced in the next section, can be considered"
"an alternative is to approximate it" probably should be "our approach is to approximate it"


LEMONADE: LEARNED MOTIF AND NEURONAL ASSEMBLY DETECTION IN CALCIUM IMAGING VIDEOS
Neuronal assemblies, loosely defined as subsets of neurons with reoccurring spatiotemporally coordinated activation patterns, or "motifs", are thought to be building blocks of neural representations and information processing. We here propose LeMoNADe, a new exploratory data analysis method that facilitates hunting for motifs in calcium imaging videos, the dominant microscopic functional imaging modality in neurophysiology. Our nonparametric method extracts motifs directly from videos, bypassing the difficult intermediate step of spike extraction. Our technique augments variational autoencoders with a discrete stochastic node, and we show in detail how a differentiable reparametrization and relaxation can be used. An evaluation on simulated data, with available ground truth, reveals excellent quantitative performance. In real video data acquired from brain slices, with no ground truth available, LeMoNADe uncovers nontrivial candidate motifs that can help generate hypotheses for more focused biological investigations.

The paper proposes a VAE-style model for identifying motifs from calcium imaging videos. As opposed to standard VAE with Gaussian latent variables it relies on Bernouli variables and hence, requires Gumbel-softmax trick for inference. Compared to methods based on matrix factorization, the proposed method has the advantage of not requiring any preprocessing on the imaging videos. My main comments are as follows:
- How sensitive is the method to the choice of beta and other hyperparameters?  Compared to SCC which has fewer hyperparameters, how robust is the method?
- How does it perform on real data compared to methods based on spike time matrices? Do they generate similar motifs?
- The application of the method seems quite limited to calcium imaging videos and it does not provide comparison with other deep generative models for videos. Methods such as Johnson et al. NIPS 2016 (Composing graphical models with neural networks for structured representations and fast inference) can also be applied to calcium imaging datasets and can potentially infer the motifs.
I believe the problem of inferring the neural motifs is an interesting problem; however, I think this paper requires more work to it shows its advantages over other deep generative models for video data and also it’s performance on real data compared to SCC (or some other matrix factorization based approach).
-----------------------------------------------------------------------
The authors have addressed my comments about other deep generative models and hyperparameter sensitivity. However, I still think the paper is more suitable for other venues with readers from the neuroscience community. Hence, I change my rating to 5.


In this work, we propose a composition/decomposition framework for adversarially training generative models on composed data -data where each sample can be thought of as being constructed from a fixed number of components. In our framework, samples are generated by sampling components from component generators and feeding these components to a composition function which combines them into a "composed sample". This compositional training approach improves the modularity, extensibility and interpretability of Generative Adversarial Networks (GANs) -providing a principled way to incrementally construct complex models out of simpler component models, and allowing for explicit "division of responsibility" between these components. Using this framework, we define a family of learning tasks and evaluate their feasibility on two datasets in two different data modalities (image and text). Lastly, we derive sufficient conditions such that these compositional generative models are identifiable. Our work provides a principled approach to building on pre-trained generative models or for exploiting the compositional nature of data distributions to train extensible and interpretable models.

The paper proposes a framework for training generative models that work on composed data. The models are trained in an adversarial fashion. The authors apply it to decompose foreground/background parts on MNIST images, and to perform sentence composition/decomposition.
High level comments:
* Clarity: In terms of language and writing style, the paper is written very clearly and easy to follow. In terms of presentation, there are some details that are omitted which would have made understanding easier and the work more reproducible.
* Quality: The idea that is introduced seems intuitive and reasonable, but the experiments does not have enough details to prove that this method works (i.e. no quantitative results presented).  Moreover, the presentation of the method is not very well done (missing details), especially since the authors used the upper limit of 10 pages.
* Originality: I am not familiar with the literature of generative models to judge this precisely, but according to the related work section it sounds like an original idea that is worth sharing.
* Significance: I believe the idea of modeling data composition explicitly sounds intuitive and interesting, and it is worth sharing. However, the experimental section does not have enough evidence that it is actually possible to learn this, so it is not clear whether the contribution is significant.
Pros:
-	interesting new problem formulation
-	simple and clear language
-	the theoretical analysis in the last section could be interesting more generally in the context of GANs
-	the framework is applied on 2 different modalities: images and text.
Cons:
-	hard to tell whether this approach works since the metrics for evaluation are not specified and there are no quantitative results in the experimental section (only 1 qualitative example per task)
-	the work is not reproducible due to the lack of details (see more explanations below)
-	the theoretical analysis is a standalone piece of the paper, without any discussion about the implications, or making connections to the previous sections.
Detailed comments:
1.	I believe the weakest part of this paper is the evaluation section. The authors run their framework on 4 tasks of increasing difficulty. While the MNIST examples make for a nice and intuitive qualitative analysis, the are no quantitative results at all. The only result that is reported for each task is one qualitative picture. The authors make statements such as “The decomposition network learns to decompose the digits and backgrounds correctly” , “Given one component, decomposition function and the other component can be learned.” but there is not mention for how these conclusion are made (no metrics, no numbers). Indeed, it is difficult in general to quantify the results of generative models, but most other GAN papers introduce some sort metric that can be used to aggregate the evaluation on an entire dataset. If the authors manually inspected the results, they should at least report how many images they inspected and how many looked correct.
2.	Aside from evaluation, there are some other details missing from the presentation. The individual details may not be major, but because all of these are missing together, it really affects the overall quality of the paper. For example:
    	 the authors state: “To train discriminator(s), a regularization is applied. For brevity, we do not show the regularization term (see Petzka et al. (2017)) used in our experiments.”. For reproducibility purposes, I believe it is important to at least mention the type of regularization, at least in the appendix.
    	There is a parameter alpha used to balance the losses. What values was used in the experiments?
    	Choices of models are often not explained. Why did you choose that form for c(o1, o2) in section 3.3? Why DCGAN for component generators, and U-net for decomposition?
    	It is not explained in detail how the Yelp-reviews dataset is altered to achieve coherence. The authors mention that “As we sample a pair independently, the input sentences are not generally coherent but the coherence can be achieved with a small number of changes.”. However, the specific algorithm by which these changes are made is not specified, and thus it can’t be reproduced.
3.	The theoretical section is an interesting contribution, but the paper just states a list of theorems without making any connections to the applications used before, or a broader discussion about how these fit in the context of GANs more generally.
4.	My understanding is that both datasets used are created by the authors by making alterations to MNIST and Yelp-reviews dataset, thus making them to some extent synthetic datasets suited to fit this problem formulation. I would have like to see how this composition/decomposition works on existing datasets with no alterations. Does it still work?
5.	In section 2.3, in the coherent sentence experimental setting, I don’t fully understand the design of the task. Figure 2 shows an example where composition and decomposition are not symmetric (i.e. composing then decomposing does not go back to the input sentences), although one of your losses is supposed to ensure exactly this cyclic consistency. Why not choose another problem that doesn’t directly violate your assumptions?
Minor issues:
6.	From the related work section, it is not clear how your approach is different from Azadi et al. (2018). Please include more details.
7.	In section 2.4, you mention using Wasserstein GANs, with no further details about this model (not even a one line description). Without reading their paper, the readers of your paper could not easily follow through this section. The losses further introduced are also not explained intuitively (e.g. what do the two expectation terms in l_g_i represent?).
8.	I believe there are some errors in which tasks reference which figures in section 3.3. Should Task 2 refers to Figure 6, and Task 3 to Figure 7?
9.	What exactly is range(.) in section 4? If this refers to the interval of values that a variable can take, the saying “is a matrix of size |range(Z)| × |range(Y )|” doesn’t exactly make sense. Please define formally.
Final remarks and advice:
Overall, I believe the paper introduces some interesting ideas. There is definitely value in the problem definition and theoretical analysis. However, I believe the paper needs more work on presentation and evaluation, especially since the authors opted for 10 pages and according to ICLR guidelines “Reviewers will be instructed to apply a higher standard to papers in excess of 8 pages.”. Hopefully the above comments will help the authors improve this work!


State-of-the-art deep neural networks are vulnerable to adversarial examples, formed by applying small but malicious perturbations to the original inputs. Moreover, the perturbations can transfer across models: adversarial examples generated for a specific model will often mislead other unseen models. Consequently the adversary can leverage it to attack deployed systems without any query, which severely hinders the application of deep learning, especially in the safety-critical areas. In this work, we empirically study how two classes of factors those might influence the transferability of adversarial examples. One is about model-specific factors, including network architecture, model capacity and test accuracy. The other is the local smoothness of loss surface for constructing adversarial examples. Inspired by these understandings on the transferability of adversarial examples, we then propose a simple but effective strategy to enhance the transferability, whose effectiveness is confirmed by a variety of experiments on both CIFAR-10 and ImageNet datasets.

The paper explores how the architecture, smoothness of the decision boundary and test accuracy of a model impacts the transferability of examples produced from it.  The paper provides a couple of novel insights, such as the asymmetry when transferring adversarial examples from one model to another. In addition, a novel method is proposed to enhance the transferability of adversarial examples from any model, through using smoothed gradients.
The experiments seem to show that the effect is rather large, and also makes the examples more robust to other transformations such as JPEG compression. Overall, these are interesting insights that could lead to further developments in making models more robust to adversarial examples. In particular, deriving adversarial examples that are both transferable and resilient to certain usual image transformations shows that the scope of the issue with adversarial examples may be even greater than what is understood today.
The paper is rather clear. Unfortunately, it is riddled with grammatical errors and should be proof-read carefully. A lot of singular/plurals are off, and some formulations are odd or downright unclear. Some examples (there are way too many to report them all):
- "Transfer-based attackS ... since they ...*
- "of adversarial exampleS ..."
- "from model A can transfer to model B"
- "less transferable than *those from* a shallow model"?
- "investigations, We ": don't capitalize
- "the averaging *has* a smoothing effect"
- "our motivation are"
- "contributed it to"
- "available *to the* adversary"
- "crafting adversarial perturbationS"
- "directly evaluation"
- "be fixed 100"
Pros:
- Transferability and robustness of adversarial examples is a very important problem
- Interesting insights, esp. the construction and evaluation of examples that are more resilient to certain image transformations
- Experimental results are convincing
Cons:
- Contribution overall may be a bit limited
- Grammatical errors and odd formulations all over the place


We reconsider the training objective of Generative Adversarial Networks (GANs) from the mixed Nash Equilibria (NE) perspective. Inspired by the classical prox methods, we develop a novel algorithmic framework for GANs via an infinite-dimensional two-player game and prove rigorous convergence rates to the mixed NE. We then propose a principled procedure to reduce our novel prox methods to simple sampling routines, leading to practically efficient algorithms. Finally, we provide experimental evidence that our approach outperforms methods that seek pure strategy equilibria, such as SGD, Adam, and RMSProp, both in speed and quality.

This paper extends the mirror-descent and mirror-prox algorithms to infinite dimensional Banach spaces so that they can be applied to solve the mixed Nash equilibrium of the popular generative adversarial networks. The main technical results appear to be formal but straightforward extensions of existing techniques in finite dimensional spaces. A sample-based practical algorithm is proposed so that the infinite dimensional algorithms can still be computed. Experiments are a bit disappointing as the authors only used visual appeal as an evaluation criterion. (I understand why the authors chose to do so but as an algorithmic paper, resorting to an evaluation based on visual appeal is almost always unsatisfactory.)
Quality: The quality of this work is moderate. Quite strangely, the authors made a fundamental mistake at the very beginning: their definition of approximate mixed equilibrium  (page 2, Notation) is bizarre and different from those in previous work (such as Nemirovski's MP paper). Fortunately, this is perhaps only an oversight on the definition; the algorithms and theorems are for the correct definition anyways. Example: consider min_{-1<= x <= 1} max_{-1<=y<=1} xy. Should we call (x, 0) an (approximate) NE for any x??
Another major issue with this work is its relaxation into mixed NE. The "bilinarization" trick in Eq (5) goes back to Kantorovich (who perhaps deserves to be mentioned), and is a relaxation in general: we now have to use a mixture of generators. Since MD/MP is not sparse, in the end we must use a large number of mixtures of generators. This certainly will create some computational issues, and make comparison to pure NE methods unfair.
Clarity: The writing of this work is mostly easily to follow. However, the presentation of the technical results suffers from a real dilemma: On one hand, the authors completely ignored the technical difference between infinite dimensional Banach spaces and finite dimensional spaces. In fact, the authors never even formally defined the underlying Banach spaces. Another example, is the mapping G on page 3 continuous? wrt what topology? without such discussion what do you mean by Frechet derivative on page 4? when is the entropy function well-defined? when is the integral of exponential well-defined? Part of me totally understand that these technicalities are daunting and perhaps should not appear in the main text. On the other hand, aren't these technicalities the only "interesting and nontrivial" part of the extension to infinite dimensional spaces? If we do not care about such technicalities and can safely "assume they can be taken care of," then why is this work nontrivial? I do not see a way to resolve this dilemma here but suggest the authors consider maybe a different venue for such type of results.
Originality: The novelty of this work is limited. The extension of MD/MP to infinite dimensional spaces is mostly formal but straightforward. In fact I believe previous authors such as Nemirovski deliberately restrict to finite dimensional spaces not because of technical incapability but to avoid uninspiring technicalities. Some very related previous works were not mentioned at all:
-- Mirror Descent Learning in Continuous Games
-- Convex Games in Banach Spaces
-- On the Universality of Online Mirror Descent
The sample based algorithms are more interesting because they make the infinite dimensional extensions implementable. However, one can not say much about their convergence behavior at the moment.
Significance: The main results, although not difficult to obtain, can potentially be very useful in broadening our arsenal of tools for training GANs. The claim "resolving the longstanding problem that no provably convergent algorithm exists for general GAN" in the Abstract is disturbing, because the authors changed the definition of GAN and because the technical contributions of this work do not live up to that strong claim.


MODEL-PREDICTIVE POLICY LEARNING WITH UNCERTAINTY REGULARIZATION FOR DRIVING IN DENSE TRAFFIC
Learning a policy using only observational data is challenging because the distribution of states it induces at execution time may differ from the distribution observed during training. We propose to train a policy by unrolling a learned model of the environment dynamics over multiple time steps while explicitly penalizing two costs: the original cost the policy seeks to optimize, and an uncertainty cost which represents its divergence from the states it is trained on. We measure this second cost by using the uncertainty of the dynamics model about its own predictions, using recent ideas from uncertainty estimation for deep networks. We evaluate our approach using a large-scale observational dataset of driving behavior recorded from traffic cameras, and show that we are able to learn effective driving policies from purely observational data, with no environment interaction.

The paper addresses the difficulty of covariate shift in model-based reinforcement learning. Here, the distribution over trajectories during is significantly different for the behaviour or data-collecting policy and the target or optimised policy. As a mean to address this, the authors propose to add an uncertainty term to the cost, which is realised by the trace of the covariance of the outputs of a MC dropout forward model. The method is applied to driving in dense traffic, where even single wrong actions can be catastrophic.
I want to stress that the paper was a pleasure to read. It was extraordinarily straightfoward to follow, because the text was well aligned with the necessary equations.
The introduction and related work seem complete to me, with two exceptions:
- Depeweg, S., Hernandez-Lobato, J. M., Doshi-Velez, F., & Udluft, S.
  (2018, July). Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning. In *International Conference on Machine Learning* (pp. 1192-1201).
- Thomas, Philip S. *Safe reinforcement learning*. Diss. University of Massachusetts Libraries, 2015.
The work by Depeweg et al addresses quite the same question as the authors of this work, but with a broader scope (i.e. not limited to traffic) but very much the same machinery. There are some important theoretical insights in this work and the connection to this submission should be drawn. In particular, the proposed method needs to be either compared to this work or it needs to be clarified why it is not applicable.
The latter appears to be of less significance in this context, but I found robust offline policy evaluation underrepresented in the related work.
I wonder if there is a way for a neural network to "hack" the uncertainty cost. I suppose that the proposed approach is an approximation to some entropy term, and it would be informative to see how exactly.
The approach shown by Eq 1 appears to be an adhoc way of estimating whether the uncertainty resulting from an action is due to the data or the model. What happens if this approach is not taken?
The objective function of the forward model is only given in the appendix. I think it needs to be moved to the main text, especially because the sum-of-squares term indicates a homoskedastic Gaussian for a likelihood. This has implications for the uncertainty estimates (see point above).
Overall, the separation of data uncertainty/risk vs model uncertainty is not done. This indicates that heterskedastic environments are candidats where the method can fail, and this limitation needs to be discussed or pointed out.
Further, the authors did not observe a benefit from using a stochastic forward model. Especially, if the prior instead of the approximate posterior is used. My point would be that, depending on the exact grapical model and the way the sampling is done to train the policy, it is actually mathematically *right* to sample from the prior. This is also how it is described in the last equation of section 2.
## Summary
Overall, I liked the paper and the way it was written. However, there are some shortcomings, such as the comparison to the work by Depeweg et al, which does a very similar thing. Also, justifying the used heuristics as approximations to a principled quantity would help. It appears that the question why and how stochastic forward models should be used requires further investigation.


Text embedding representing natural language documents in a semantic vector space can be used for document retrieval using nearest neighbor lookup. In order to study the feasibility of neural models specialized for retrieval in a semantically meaningful way, we suggest the use of the Stanford Question Answering Dataset (SQuAD) in an open-domain question answering context, where the first task is to find paragraphs useful for answering a given question. First, we compare the quality of various text-embedding methods on the performance of retrieval and give an extensive empirical comparison on the performance of various non-augmented base embedding with, and without IDF weighting. Our main results are that by training deep residual neural models specifically for retrieval purposes can yield significant gains when it is used to augment existing embeddings. We also establish that deeper models are superior to this task. The best base baseline embeddings augmented by our learned neural approach improves the top-1 recall of the system by 14% in terms of the question side, and by 8% in terms of the paragraph side.

This paper tries to study retrieval methods for multi-paragraph / multi-document reading comprehension.  The basic approach is to embed the question and the paragraph and train a system to put the correct paragraph close to the question.  I had a very hard time following the details of the proposed approach for this, however, and I still don't really understand what the authors are proposing.
This paper is not ready for publication.  The exposition is not at all clear and needs substantial rewriting.  Additionally, the evaluation done in the paper is not well-justified.  I do not know what "paragraph-side" means, but I assume that means you are trying to retrieve the question given the paragraph.  Why?  There were no standard baselines compared against, like a simple IR system (Lucene).  And I expected to see actual impact of the retrieved results on downstream QA performance of a system like Chen et al.'s, or Clark and Gardner 2018.  Even if you have a slightly better ranking of the retrieved paragraphs, it's not clear to me that this will improve performance, if the downstream method is properly calibrated to handle multiple paragraphs (see Clark and Gardner 2018).
A few writing suggestions for the authors, for next time:
This paper does not follow the typical flow of an academic paper.  It reads too much like a logbook of what you did, presented chronologically, instead of presenting the ideas in a coherent sequence.  Part of this is just simple wording fixes (e.g., avoid things like "it was time to compute ELMo representations" - this isn't a logbook).  Also, all of the shape comments and numerical details at the top of page 4 are out of place.  Describe your method first in general terms, then give experimental details (like corpus size, etc.) later.  I suggest reading the award-winning papers at various conferences to get a sense of how these papers are typically structured and phrased.
Section 2: A full page dedicated to the history of word embeddings is entirely unnecessary for this paper.  This is not a survey on word embeddings.  It's much more useful to the reader to give pointers to multiple connection points between your work and the rest of the literature.  You could have given a paragraph to the most relevant embedding techniques, a paragraph to the most relevant retrieval / multi-paragraph techniques (e.g., Clark and Gardner 2018, which is very relevant, along with Chen et al., TriviaQA, others), and a paragraph to distance metric learning.


Various NLP problems -such as the prediction of sentence similarity, entailment, and discourse relations -are all instances of the same general task: the modeling of semantic relations between a pair of textual elements. We call them textual relational problems. A popular model for textual relational problems is to embed sentences into fixed size vectors and use composition functions (e.g. difference or concatenation) of those vectors as features for the prediction. Meanwhile, composition of embeddings has been a main focus within the field of Statistical Relational Learning (SRL) whose goal is to predict relations between entities (typically from knowledge base triples). In this work, we show that textual relational models implicitly use compositions from baseline SRL models. We show that such compositions are not expressive enough for several tasks (e.g. natural language inference). We build on recent SRL models to address textual relational problems, showing that they are more expressive, and can alleviate issues from simpler compositions. The resulting models significantly improve the state of the art in both transferable sentence representation learning and relation prediction.

This paper presents a view of sentence-level prediction tasks as statistical relation learning problems. In particular the paper argues that composition functions used in recent SRL techniques developed for entity-to-entity relationship detection can be applied to sentence-level relation prediction tasks.
Suppose there is a prediction training task defined over pairs of sentences (x1, x2). This task requires some function 'f' that composes the sentence representations h1 and h2 into a single representation which is then used to
make the relation prediction i.e., we have a model g(f(h1, h2)) that is used to predict some relation between R(x1, x2).  This paper aims to show that with a better 'f' we can hope for a better result in transfer tasks (in addition to doing better on the training task).
The paper argues that this setting, at a high level, is similar to the composition function used in entity-entity relation prediction. There have been many such methods in the recent past (e.g., TransE, ComplEx, RESCAL). This paper asks whether these composition functions can work well for sentence-level tasks.
The paper then presents experiments which compare the performance of different composition functions against a basic composition function used in InferSent.
Strengths of the paper:
1. I like the main question of what can we learn from SRL. This seeks to bridge some independent research threads.
2. The evaluation considers a range of composition functions used in SRL and applies them to the sentence tasks.
3. Points out that some of the composition functions used in existing models are not particularly strong.
Issues:
I like the starting point for this paper very much and agree that the existing composition functions for sentence relations are rather weak. However, I am struggling to see if there is (i) a convincing conceptual argument for why SRL view of compositions is necessarily the answer for sentence level tasks, or (ii) a convincing empirical case for the same.  Some details on these points:
1) The parallels between entity-entity relations and sentence-sentence relations seems a bit of a stretch to me. There is always some level of abstraction at which two problems might look similar, which can be advantageous for repurposing solutions. However, in this case I think the SRL view of the world hides the complexities in sentence-sentence relation tasks (e.g. aligning relevant pieces of information, requiring more complex composition functions to derive meaning etc.).
2) I am not sure what knowledge we are getting from an SRL view of the problem that is not already known already to the communities that work on sentence embedding. The minimum requirements laid out can be met easily by existing methods for sentence representations. For instance that we need to allow for asymmetric relations (entailment order) is very well known. As the authors themselves point out there are solutions for this problem.
3) The empirical results don't appear convincing. The average gain for any particular method over InferSent is 0.3 in macro average. There is no single SRL based composition method that works consistently clear gains across most tasks.
Here are some suggestions that I think will improve the paper (or at least help me buy the motivation):
1. One question that might be useful to make a conceptual argument is how much work should be done in 'f' and should it change for the different type of target tasks.
If the idea is to transfer h for single sentence target task, then a powerful 'f' can render h1 and h2 to be simple enough, such that bulk of the work in extracting task related information might be done by 'f' itself. Therefore, transferred h may not be as powerful as it could have been with a less powerful 'f'.
If the idea is to transfer f(h1, h2) for sentence-pair target tasks, then a powerful 'f' might be a good thing.
2) Another useful discussion would be to discuss why more powerful alignment based sentence representations are not being considered at least for comparison purposes.
The paper wants to go from a simple 'f' (i.e. concat(h1,h2), h1-h2) to some other choices for 'f' that are known functions from SRL.
There are several sentence-level representation functions such as ESIM [Chen et al., 2016] which uses a combined representation of premise and hypothesis sentences using soft alignment to specifically address the issues in comparing sentences. A similar representation is computed in BiDAF [Seo et al., 2017] in the context of matching question representation with sentence representations.
To summarize, I really like the basic starting point for the paper and would love to see a more compelling presentation of the conceptual argument and a stronger empirical comparison.