Reparameterization of variational auto-encoders is an effective method for reducing the variance of their gradient estimates. However, when the latent variables are discrete, a reparameterization is problematic due to discontinuities in the discrete space. In this work, we extend the direct loss minimization technique to discrete variational auto-encoders. We first reparameterize a discrete random variable using the arg max function of the Gumbel-Max perturbation model. We then use direct optimization to propagate gradients through the non-differentiable arg max using two perturbed arg max operations.

This paper proposes combining the Gumbel-max trick and "direct loss optimization" for variance reduction in VAEs with discrete latent variables. This is a natural combination (in hindsight), since the Gumbel-max trick turns sampling into non-differentiable optimization, and direct loss optimization provides a way to optimize the expected value of a non-differentiable loss. The paper is well-written for the most part and is backed by good experimental results. However it like some of the mathematical details and some of the exposition could be greatly improved.
I think there are several mistakes in the reasoning presented in the proof of Theorem 1 (see detailed comments below). Theorem 1 in the current paper seems to me to be a special case of Theorem 1 in (Song 2016), where the expectation over data is replaced by an expectation over the Gumbel variable gamma. If I've understood correctly, it seems like it would be more correct and concise to simply cite that paper with some explanatory comments.
The word "direct" occurs quite a lot in the paper. It sometimes seemed misplaced. For example for "The direct differentiation of the resulting expectation" in the introduction, in what sense is the differentiation direct, and what would non-direct differentiation be?
In section 3, that's not the meaning of the term "exponential family".
The re-use of theta and psi as both model parameters and the log probability density / distribution is unnecessarily confusing.
A small point, but in "the challenge in generative learning is to reparameterize and optimize (2)", the authors assume that q has analytic expression for the second KL term in (1). That's often the case but definitely not always. Also, even if this KL term has an analytic expression, it is not always better to use it (see Duvenaud "Sticking the landing...").
In (3), the usual notation is P(x = i) where x is the random variable and i is its possible value, whereas in (3) the random variable z^{\phi + \gamma} appears on the right of the equals sign.
The first paragraph of section 4.1 and (4) and (5) are just a simple application of the law of total expectation, and it would be simpler and clearer to state that.
"gradient of the decoder" should be "gradient of the decoder log probability" (or log prob density depending on preference). Similarly with "the decoder is a smooth function". The decoder is a conditional probability distribution (at least according to my understanding of conventional usage).
In the first equation in the proof of Theorem 1, it seems as though the authors are using the standard change of variables formula for integrals. However the new variable \hat{\gamma} depends on \hat{z} through \theta, so I don't see how it's valid to ignore the max in the way the present paper does. One way to see that something is wrong is the fact that the integrand on LHS has \hat{z} as a bound variable only, whereas the integrand on RHS has \hat{z} as both a bound variable (inside the max) and a free variable (since \theta depends on \hat{z}, though strangely that is not written in the equation). What is the value of \hat{z} used for \theta on RHS?
There's a missing [] after \partial_\epsilon in the third line of the paragraph starting "We turn to prove Equation (8)".
In the same line, I don't see why the two expectations are equal. It seems to me that the differentiation w.r.t. epsilon ignores the fact that changing epsilon occasionally changes z^{\epsilon \theta + \phi_v + \gamma} in a discontinuous way. The term being differentiated has both a continuous-in-epsilon component and a piecewise-constant-in-epsilon component, and the latter appears to have been ignored. While the gradient of a piecewise constant function is zero almost everywhere, the occasional large changes (which could be thought of as delta functions) still can make a large contribution to the overall expression once we take the expectation. To look at it another way, if the reasoning here is correct, why can't the same argument be used on the RHS of (8), first to take the derivative inside the expectation and subsequently to compute the derivative as zero, since the inner term is a piecewise constant function of v? Yet clearly the RHS of (8) is not always zero.
Around "However when we approach the limit, the variance of the estimate increases...", I think it would be extremely helpful to explain that for small epsilon, we occasionally obtain a large gradient (and otherwise zero), while for large epsilon we often obtain a moderate non-zero gradient. That gives some insight into the effect of epsilon, and why the variance is larger for small epsilon.
Any reason not to plot the bias in right Figure 1, which is ostensibly about the bias-variance trade-off?
I didn't follow the meaning of the diagram or caption for left Figure 1.
In (11), I wasn't sure whether S included the supervised examples or not (i.e. whether S_1 was disjoint from or a subset of S). If disjoint, shouldn't the KL term be included, or the expectation-over-gamma term be changed to use ground truth z? I guess I was unclear on the form of loss used for the supervised data, and unclear on the motivation for this choice.
In the last sentence of section 5.1, should "chain rule" be "variance reparameterization trick"?
In section 5.2, it would be helpful to mention what mean field means in terms of the variational distribution q (namely q(z | x) = \prod_i q(z_i | x) ). Also, the term "mean field" is not conventionally used for general distributions (such as the decoder here) as far as I'm aware, only for variational distributions. "Conditionally independent" might be clearer.
What does "for which we can approximate z^{...} efficiently" refer to?
In section 6.1, what is the annealing rate? Also, the minimal epsilon is set to 0.1. Is epsilon changed as training progresses according to some schedule?
"The main advantage of our framework is that it seamlessly integrates semi-supervised learning" seems like an overstatement. Wouldn't semi-supervised learning be relatively straightforward to incorporate into any form of VAE? And why not just use log p(x, z) for updating the decoder parameters and log q(z | x) for updating the encoder parameters?
How many labeled examples were used for the CelebA semi-supervised learning?
Some bibliography typos. For example, no capitalization throughout (e.g. "gumbel" instead of "Gumbel"). Also lots of arxiv preprints cited when published papers exist (e.g. Jang 2017 should be ICLR 2017 not arxiv preprint).


Multi-relational graph embedding which aims at achieving effective representations with reduced low-dimensional parameters, has been widely used in knowledge base completion. Although knowledge base data usually contains tree-like or cyclic structure, none of existing approaches can embed these data into a compatible space that in line with the structure. To overcome this problem, a novel framework, called Riemannian TransE, is proposed in this paper to embed the entities in a Riemannian manifold. Riemannian TransE models each relation as a move to a point and defines specific novel distance dissimilarity for each relation, so that all the relations are naturally embedded in correspondence to the structure of data. Experiments on several knowledge base completion tasks have shown that, based on an appropriate choice of manifold, Riemannian TransE achieves good performance even with a significantly reduced parameters.

The paper proposes a new approach to compute embeddings of multi-relational data such as knowledge graphs. For this purpose, the paper introduces a variant of TransE that operates on Riemannian manifolds, in particular, Euclidean, Spherical, and Hyperbolic space. This approach is motivated by the results of Nickel & Kiela (2017), who showed tha  Hyperbolic space can provide important advantages for embedding graphs with hierarchical structure.
Hyperbolic and Riemannian embeddings are a promising research area that fits well into ICLR. Extending hyperbolic, and more general, Riemannian embeddings to multi-relational data is an important aspect in this context, as it allows to extend such methods to new applications such as Knowledge Graph Completion. Overall, the paper is written well and mostly good to understand. However, I am concerned about multiple aspects in the current version:
- What is the motivation for using this particular form of translation? In Riemannian manifolds, the analogue of vector addition and subtraction is typically taken as the exponential or logarithmic map (as expm and logm in Euclidean space are exactly vector addition and subtraction). For the spherical and hyperbolic manifold both maps have closed form expressions and are differentiable. It is therefore not clear to me what the advantage of the proposed approach is compared to these standard methods. In any case, it would be important to include them in the experimental results.
- It is also not clear to me whether the benefits of hyperbolic embeddings translate into the setting that is proposed here. The advantage of hyperbolic embeddings is that they impose a hierarchical structure in the latent space. The method proposed in this paper uses then a single (hyperbolic) embedding of entities across all relation types. This implies that there should be a single consistent hierarchy that explains all the links in all relations. This seems unlikely and might explain some of the hyperbolic results. A more detailed discussion and motivation would be important here.
- Regarding the experimental results: Why are the results for HolE and ComplEx so different? As [1,2] showed, both models are identical, and for that reason should get close to identical results. The large differences seem inconsistent with these results. Furthermore it seems that the results reported in this paper do not match previously reported results. What is the reason for these discrepancies?
[1] Hayashi, K., and Shimbo, M. "On the equivalence of holographic and complex embeddings for link prediction", 2017.
[2] Trouillon, T, and Nickel, M. "Complex and Holographic Embeddings of Knowledge Graphs: A Comparison", 2017.


STRUCTURED ADVERSARIAL ATTACK: TOWARDS GENERAL IMPLEMENTATION AND BETTER INTERPRETABILITY
When generating adversarial examples to attack deep neural networks (DNNs), p norm of the added perturbation is usually used to measure the similarity between original image and adversarial example. However, such adversarial attacks perturbing the raw input spaces may fail to capture structural information hidden in the input. This work develops a more general attack model, i.e., the structured attack (StrAttack), which explores group sparsity in adversarial perturbations by sliding a mask through images aiming for extracting key spatial structures. An ADMM (alternating direction method of multipliers)-based framework is proposed that can split the original problem into a sequence of analytically solvable subproblems and can be generalized to implement other attacking methods. Strong group sparsity is achieved in adversarial perturbations even with the same level of p -norm distortion (p ∈ {1, 2, ∞}) as the stateof-the-art attacks. We demonstrate the effectiveness of StrAttack by extensive experimental results on MNIST, CIFAR-10 and ImageNet. We also show that StrAttack provides better interpretability (i.e., better correspondence with discriminative image regions) through adversarial saliency map (Papernot et al., 2016b) and class activation map (Zhou et al., 2016). Our code is available at https://github.com/KaidiXu/StrAttack.

The paper proposes a novel approach to generate adversarial examples based on structured sparsity principles. In particular the authors focus on the intuition that adversarial examples in computer vision might benefit from encoding information about the local structure of the data. To this end, lp *group* norms can be used in contrast to standard global lp norms when constraining or penalizing the optimization of the adversarial example. The authors propose an optimization strategy to address this problem. The authors evaluate the proposed approach on real data, comparing it against state-of-the-art competitors, which do not leverage the structured sparsity idea.
The paper is well written and easy to follow. The presentation of the algorithms for i) the non-overlapping and ii) overlapping groups as well as iii) the proposed refinement are clear. The experimental evaluation is interesting and convincing (the further experiments in the supplementary material add value to the overall discussion).
The main downside of the paper is that the proposed idea essentially consists in replacing the standard \ell_p norm penalty/constraints with a group-\ell_p one. While this provides interesting technical questions from the algorithmic perspective, from the point of view of the novelty, the paper does not appear an extremely strong contribution,


Word embeddings are known to boost performance of many NLP tasks such as text classification, meanwhile they can be enhanced by labels at the document level to capture nuanced meaning such as sentiment and topic. Can one combine these two research directions to benefit from both? In this paper, we propose to jointly train a text classifier with a label-enhanced and domain-aware word embedding model, using an unlabeled corpus and only a few labeled data from non-target domains. The embeddings are trained on the unlabed corpus and enhanced by pseudo labels coming from the classifier, and at the same time are used by the classifier as input and training signals. We formalize this symbiotic cycle in a variational Bayes framework, and show that our method improves both the embeddings and the text classifier, outperforming state-of-the-art domain adaptation and semisupervised learning techniques. We conduct detailed ablative tests to reveal gains from important components of our approach. The source code and experiment data will be publicly released.

The authors propose a semi-supervised learning techhnique involving jointly tuning word embeddings and a classifier. The idea is to rapidly adapt models to new domains with minimal (actually, zero) supervision by exploiting such embeddings.
In effect, this is an approach to "disentangling" sentiment from domain, via a variational objective and semi-supervision via the embedding parameters. This is a nice idea as disentanglement enables transfer (or should). The experiments are well-executed (albeit on a limited set of classification tasks). Still the comparisons are relatively exhaustive, and the authors have done a nice job of providing ablations. I found Table 4 and Figure 2 particularly nice.
- The authors should speak to the generalizability of this approach beyond sentiment tasks, as presently it seems largely constrained to this domain. This would of course hinder the potential utility/impact of the work.
- Due to the very concise description of baselines, it was not obvious to me if these were also taking a semi-supervised approach? I *think* the pivot-based model is. But then how was CNN+ELMo trained exactly? With zero target sentiment labels, I presume? If so, an obvious baseline would be to "pseudo-label" instances in the target domain first, and then use these predictions as a target to fine-tune the CNN (or whatever), back-propping through to the embeddings. Was this done? In general more details on the baselines and training setups should be included, even if only in the Appendix.
- The strategy, I think, is to learn a prediction model p_\phi(y|z, D) where \phi are model parameters. Then this is applied to instances in the target dataset to infer latent z, and then use these inferred labels to fine-tune word embeddings. Specifically, sentiment label information is pushed into embeddings via a modified CBOW objective that effectively shifts the meaning (as codified in the distributed representation) via task specific sentiment. One thing here that confused me is that this would seem to perform this affine transformation to all words, but only certain of these will exhibit domain specific variation with respect to sentiment. Can the authors speak to this?
- I found Eq 7 counterintuitive, since it treats sentiment and domain independently, but the authors explicitly noted above that this is not the case, i.e., words will depend jointly on sentiment + domain. I actually think p(w_i|y) also depends on the domain via M (implicitly), but perhaps this should be made explicit (or perhaps I am misunderstanding something!)
- In 2.1, the authors assume a prior p(z|c) and then say that `'naively' marginalizing over this performs poorly. But I not think the particular prior distribution used was not discussed here. Could they elaborate?
- I think i indices are needed on the w's in Eq 6?


Neural networks are known to be a class of highly expressive functions able to fit even random input-output mappings with 100% accuracy. In this work we present properties of neural networks that complement this aspect of expressivity. By using tools from Fourier analysis, we show that deep ReLU networks are biased towards low frequency functions, meaning that they cannot have local fluctuations without affecting their global behavior. Intuitively, this property is in line with the observation that over-parameterized networks find simple patterns that generalize across data samples. We also investigate how the shape of the data manifold affects expressivity by showing evidence that learning high frequencies gets easier with increasing manifold complexity, and present a theoretical understanding of this behavior. Finally, we study the robustness of the frequency components with respect to parameter perturbation, to develop the intuition that the parameters must be finely tuned to express high frequency functions.

Summary.
This paper has theoretical and empirical contributions on topic of Fourier coefficients of neural networks.  First is upper bound on Fourier coefficients in terms of number of affine pieces and Lipschitz constant of network.  Second is collection of synthetic data and trained networks whereupon is argued that neural networks focus early effort upon low Fourier coefficients.
Brief evaluation.
Pros:
+ This paper attacks important and timely topic: identifying and analyzing implicit bias of neural networks paired with standard training methods.
Cons:
- "Implicit bias" hypothesis has been put forth by many authors for many years, and this paper does not provide compelling argument that Fourier coefficients provide good characterization of this bias.
- Regarding "many authors for many years", this paper fails to cite and utilize vast body of prior work, as detailed below.
- Main theorem here is loose upper bound primarily derived from prior work, and no lower bounds are given.  Prior work does assess lower bounds.
- Experiments are on synthetic data; prior work on implicit regularization does check real data.
Detailed evaluation.
* "Implicit bias" hypothesis appears in many places, for instance in work of Nati Srebro and colleagues ("The Implicit Bias of Gradient Descent on Separable Data" (and follow-ups), "Exploring generalization in deep learning" (and follow-ups), and others); it can also be found in variety of recent generalization papers, for instance again the work of Srebro et al, but also Bartlett et al, Arora et al.  E.g., Arora et al do detailed analysis of favorable biases in order to obtain refined generalization bound.  Consequently I expect this paper to argue to me, with strong theorems and experiments, that Fourier coefficients are a good way to assess implicit bias.
* Theorem 1 is proved via bounds and tools on the Fourier spectra of indicators of polytopes due to Diaz et al, and linearity of the Fourier transform.  It is only upper bound (indeed one that makes no effort to deal with cancellations and thus become tight).  By contrast, the original proofs of depth separation for neural networks (e.g., Eldan and Shamir, or Telgarsky, both 2015), provide lower bounds and metric space separation.  Indeed, the work of Eldan&Shamir extensively uses Fourier analysis, and the proof develops a refined understanding of why it is hard for a ReLU network to approximate a Fourier transform of even simple functions: it has to approximate exponentially many tubes in Fourier space, which it can only do with exponentially many pieces.  While the present paper aims to cover some material not in Eldan&Shamir --- e.g., the bias with training --- this latter contribution is argued via synthetic data, and overall I feel the present work does not meet the (high) bar set by Eldan&Shamir.
*  I will also point out that prior work of Barron, his "superposition" paper from 1993, is not cited. That paper presents upper bounds on approximation with neural networks which depends on the Fourier transform.  There is also follow-up by Arora et al with "Barron functions".
* For experiments, I would really like to see experiment showing Fourier coefficients at various stages of training of standard network on standard data and standard data but with randomized labels (or different complexity in some other way).  These Fourier coefficients could also be compared to other "implicit bias" quantities; e.g., various norms and complexity measures.  In this way, it would be demonstrated that (a) spectral bias happens in practice, (b) spectral bias is a good way of measuring implicit bias.  Admittedly, this is computationally expensive experiment.
* Regarding my claim that Theorem 1 is "loose upper bound": the slope of each piece is being upper bounded by Lipschitz constant, which will be far off in most regions.  Meanwhile, Lemma 1, "exact characterization", does not give any sense of how the slopes relate to weights of network.  Improving either issue would need to deal with "cancellations" I mention, and this is where it is hard to get upper and lower bounds to match.
I feel this paper could be made much stronger by carefully using the results of all this prior work; these are not merely citation omissions, but indeed there is good understanding and progress in these papers.


CEM-RL: COMBINING EVOLUTIONARY AND GRADIENT-BASED METHODS FOR POLICY SEARCH
Deep neuroevolution and deep reinforcement learning (deep RL) algorithms are two popular approaches to policy search. The former is widely applicable and rather stable, but suffers from low sample efficiency. By contrast, the latter is more sample efficient, but the most sample efficient variants are also rather unstable and highly sensitive to hyper-parameter setting. So far, these families of methods have mostly been compared as competing tools. However, an emerging approach consists in combining them so as to get the best of both worlds. Two previously existing combinations use either an ad hoc evolutionary algorithm or a goal exploration process together with the Deep Deterministic Policy Gradient (DDPG) algorithm, a sample efficient off-policy deep RL algorithm. In this paper, we propose a different combination scheme using the simple cross-entropy method (CEM) and Twin Delayed Deep Deterministic policy gradient (TD3), another off-policy deep RL algorithm which improves over DDPG. We evaluate the resulting method, CEM-RL, on a set of benchmarks classically used in deep RL. We show that CEM-RL benefits from several advantages over its competitors and offers a satisfactory trade-off between performance and sample efficiency.

Gradient-free evolutionary search methods for Reinforcement Learning are typically very stable, but scale poorly with the number of parameters when optimizing highly-parametrized policies (e.g. neural networks). Meanwhile, gradient-based deep RL methods, such as DDPG are often sample efficient, particularly in the off-policy setting when, unlike evolutionary search methods, they can continue to use previous experience to estimate values. However, these approaches can also be unstable.
This work combines the well-known CEM search with TD3 (an improved variant of DDPG). The key idea of of this work is in each generation of CEM, 1/2 the individuals are improved using TD3 (i.e. the RL gradient). This method is made more practical by using a replay buffer so experience from previous generations is used for the TD3 updates and importance sampling is used to improve the efficiency of CEM.
This work shows, on some simple control tasks, that this method appears to result in much stronger performance compared with CEM, and small improvements over TD3 alone. It also typically out-performs ERL.
Intuitively, it seems like it may be possible to construct counter-examples where the gradient updates will prevent convergence. Issues of convergence seem like they deserve some discussion here and potentially could be examined empirically (is CEM-TD3 converging in the swimmer?).
The justification that the method of Khadka & Tumer (2018) cannot be extended to use CEM, since the RL policies do not comply with the covariance matrix is unclear to me. Algorithm 1, step 20, the covariance matrix is updated after the RL step so regardless of how the RL policies are generated, the search distribution on the next distribution includes them. In both this work, and Khadka & Tumer, the RL updates lead to policies that differ from the search distribution (indeed that is the point), and there is no guarantee in this work that the TD3 updates result in policies close to the starting point. It sees like the more important distinction is that, in this approach, the information flows both from ES to RL and vice-versa, rather than just from RL to ES.
One view of this method would be that it is an ensemble method for learning the policy [e.g. similar to Osband et al., 2016 for DQN]. This could be discussed and a relevant control would be to keep a population (ensemble) of policies, but only update using RL while sharing experience across all actors. This would isolate the ensemble effect from the evolutionary search.
Minor issues:
- The ReLU non-linearity in DDPG and TD3 prior work is replaced with tanh. This change is noted, but it would be useful to make a least a brief (i.e. one sentence) comment on the motivation for this change.
- The paper is over the hard page limit for ICLR so needs to be edit to reduce the length.
Osband I, Blundell C, Pritzel A, Van Roy B. Deep exploration via bootstrapped DQN. InAdvances in neural information processing systems 2016 (pp. 4026-4034).


We propose a segmental neural language model that combines the representational power of neural networks and the structure learning mechanism of Bayesian nonparametrics, and show that it learns to discover semantically meaningful units (e.g., morphemes and words) from unsegmented character sequences. The model generates text as a sequence of segments, where each segment is generated either character-by-character from a sequence model or as a single draw from a lexical memory that stores multi-character units. Its parameters are fit to maximize the marginal likelihood of the training data, summing over all segmentations of the input, and its hyperparameters are likewise set to optimize held-out marginal likelihood. To prevent the model from overusing the lexical memory, which leads to poor generalization and bad segmentation, we introduce a differentiable regularizer that penalizes based on the expected length of each segment. To our knowledge, this is the first demonstration of neural networks that have predictive distributions better than LSTM language models and also infer a segmentation into word-like units that are competitive with the best existing word discovery models.

[Note to the authors: I was assigned this paper after the reviewing deadline.]
The authors train language models on unsegmented text, simultaneously discovering word boundaries
without direct supervision. Given the past history, but ignoring past segmentation decisions
to keep computations tractable, the model predicts the next character segment (word-like unit)
by combining a character-level LSTM with a lexical memory. To prevent overusing the
lexical memory, which would lead to poor generalization, the authors propose a segment
length penalty.
Strengths:
The model architecture is interesting, combining the benefits of a character-level
model (open vocabulary) with those of a lexical model (effective for frequent character
sequences).
Despite the exponential number of possible segmentations, inference remains tractable
using dynamic programming (with some simplifying assumptions).
The ablation study clearly shows that both the lexical memory and the length penalty
contribute significantly.
Weaknesses:
The writing quality is somewhat weak. Many errors should have been caught when
proofreading the paper (e.g. "The segmentation decisions and decisions" and
"The the characters" on page 1).
I am confused by the key-value pairs of the lexical memory. Shouldn't character
sequences be keys, and their trainable vector representations be values?
It is hard to evaluate how good the language models are, as the strength of the
baselines is unclear. How well-tuned is the LSTM?
Comparison to some other segmentation approaches (not necessarily with language modeling)
is limited. In particular, adaptor grammars perform very well on the Brent corpus [1].
However, [2] is mentioned briefly. As these other approaches work better for segmentation,
the authors should carefully justify why having a single model that does both language
modeling and word segmentation well matters. Many neural approaches have also been
suggested for Chinese word segmentation (among others [3]). In these papers, results on the
PKU dataset are much better.  Are these directly comparable with yours?
I would have liked a finer analysis of the impact of the length penalty.
A plot showing how validation likelihood and segmentation performance vary as
\lambda is increased could potentially be interesting.
[1] Johnson and Goldwater. "Improving nonparameteric Bayesian inference: experiments on
unsupervised word segmentation with adaptor grammars", HLT, 2009
[2] Berg-Kirkpatrick et al. "Painless Unsupervised Learning with Features", NAACL, 2010
[3] Yang et al. Yang, Jie, Yue Zhang, and Fei Dong. "Neural Word Segmentation with Rich Pretraining", ACL, 2017


Many real-world problems, e.g. object detection, have outputs that are naturally expressed as sets of entities. This creates a challenge for traditional deep neural networks which naturally deal with structured outputs such as vectors, matrices or tensors. We present a novel approach for learning to predict sets with unknown permutation and cardinality using deep neural networks. Specifically, in our formulation we incorporate the permutation as unobservable variable and estimate its distribution during the learning process using alternating optimization. We demonstrate the validity of this new formulation on two relevant vision problems: object detection, for which our formulation outperforms state-of-the-art detectors such as Faster R-CNN and YOLO, and a complex CAPTCHA test, where we observe that, surprisingly, our set based network acquired the ability of mimicking arithmetics without any rules being coded.

— Summary
The method extends [21], which proposes an unordered set prediction model for multi-class classification. For that problem, [21] can assume logistic outputs for all distinct classes. This work extends set prediction to the object detection task, where box identity is not distinct — this is handled by an additional model output that reasons about the most likely object permutations. The permutation predictions are used during training, but are not needed at inference time — as shown in Fig1 and Eq 7. Results are on detection of overlapping objects and a CAPTCHA toy summation example.
— Clarity
The exposition is not particularly clear in several places:
 - U^m in Eq 1 is undefined and un-discussed. What probability term does it correspond to? It is supposed to make probabilities of different cardinalities comparable, but the exact mechanism is unclear.
 - The term p(w) disappears on the left hand side of Eq 2.
 - Notation in Sec. 3.2 is very cumbersome, making it hard to follow. Furthermore, I found the description ambiguous, preventing me from understanding how exactly the permutation head output is used in Eq 5. Specifically, there is some confusion about estimation of w~, which seems based on frequency estimation from past SGD iterations (Eq 3). If so, why does term f2 in Eq 5 contain the permutation head output O2 and how do the two relate?
 - The network architecture is never described, especially the transition from Conv to Dense and the layer sizes, making the work hard to reproduce. The dimensions of the convolutional feature map matter (probably need to be kept tractable).
— Significance
Key aspects of the model are not particularly clear, specifically about how the permutation prediction ( the key novelty here) is used to benefit training.
— Term f2 in Eq5 uses w~ estimates, which appeared to be based on statistics from past SGD runs, yet also depends on the output of the permutation head O2. Am I misinterpreting the method?
— In the paragraph right after Eq5, it’s claimed that “Empirically, in our applications, we found out that estimation of the permutations from just f1 [in Eq5] is sufficient to train properly … by using the Hungarian algorithm”. So then f2 term is not even used in. Eq5? If so, what is the significance of the permutation head other than adding an auxiliary loss?
Furthermore, there are no experimental results demonstrating the effect of the permutation head and the design choices above — if we could get by with only using the Hungarian algorithm, why bother classifying an exponential number of permutations? Do they help when added as an auxiliary loss?
While the failure of NMS to detect overlapping objects is expected, the experiments showing that perm-set prediction handles them well is interesting and promising. Solving the general case with larger images and many instances would increase the impact significantly — and likely require a combination of perm-set prediction and image tiling, although this is just a hypothesis. The Captcha toy example also shows some interesting behavior emerging — without digit-specific annotations (otherwise it would be multi-class classification setup from [21]), the model can handle the majority of summations correctly.
— Experimental results
The results are interesting proofs-of-concept but a few more experiments/answers would be helpful:
- It still appears that PR curve in the high-precision regime (fig 3b) has lower precision than FRCNN/YOLO. Any idea as to why?
- Ablation results on the effect of the permutation predictions vs Hungarian algorithm, etc would be helpful, as discussed above.
- How sensitive is the method to seeing a certain cardinality? What if it never sees 3 pedestrians in an image, but only 1,2,4 will it fail to predict 3? Or alternatively, if we train a model that can handle up to 5-6 entities with examples than have <=4? What is the right way of data augmentation for this model (was there any and should there be?)
- Given that values for U differ across applications, how sensitive is the output / how much sweeping did you have to do?

-- Related work
To the best of my knowledge it's representative. It would help to cite more recent work that decreases detector dependence on NMS. For example, "Learning Non-Maximum Suppression", Hosang, Benenson, Schiele, CVPR 2017 or "Relation Networks for Object Detection", by Hu et al, CVPR 2018 and references therein.


WHITENING AND COLORING BATCH TRANSFORM FOR GANS
Batch Normalization (BN) is a common technique used to speed-up and stabilize training. On the other hand, the learnable parameters of BN are commonly used in conditional Generative Adversarial Networks (cGANs) for representing classspecific information using conditional Batch Normalization (cBN). In this paper we propose to generalize both BN and cBN using a Whitening and Coloring based batch normalization. We show that our conditional Coloring can represent categorical conditioning information which largely helps the cGAN qualitative results. Moreover, we show that full-feature whitening is important in a general GAN scenario in which the training process is known to be highly unstable. We test our approach on different datasets and using different GAN networks and training protocols, showing a consistent improvement in all the tested frameworks. Our CIFAR-10 conditioned results are higher than all previous works on this dataset.

This paper proposed Whitening and Coloring (WC) transform to replace batch normalization (BN) in generators for GAN. WC generalize BN by normalizing features with decorrelating (whitening) matrix, and then denormalizing (coloring) features by learnable weights. The main advantage of WC is that it exploits the full correlation matrix of features, while BN only considers the diagonal. WC is differentiable and is only 1.32x slower than BN. The authors also apply conditional WC, which learn the parameters of coloring conditioned on labels, to conditional image generation.  Experimental results show WC achieves better inception score and FI distance comparing to BN on CIFAR-10, CIFAR-100, STL-10 and Tiny Imagenet. Furthermore, the conditional image generation results by WC are better than all previous methods.
I have some detailed comments below.
+ The paper is well written, and I generally enjoyed reading the paper.
+ The experimental results look sufficient, and I appreciate the ablation study sections.
+ The score on supervised CIFAR-10 is better than previous methods.
- The main text is longer than expectation. I would suggest shorten section 3.1 Cholesky decomposition, section 4 conditional color transformation and the text in section 5 experiments.
- The proposed WC transform is general. It is a bit unclear why it is particularly effective for generator in GAN. Exploiting the full correlation matrix sounds reasonable, but it may also introduce unstability. It would help if the authors have an intuitive way to show that whitening is better than normalization.
- It is unclear why conditional WC can be used for generation conditioned on class labels. In Dumoulin 2016, conditional instance normalization is used for generating images conditioned on styles. As image styles are described by Gram matrix (correlation) of features, changing first order and second order statistics of features is reasonable for image generation conditioned on styles. I cannot understand why conditional WC can be used for generation conditioned on class labels. I would like the authors to carefully explain the motivation, and also provide visual results like using the same random noise as input, but only changing the class conditions.
- It is unclear to me why the proposed whitening based on Cholesky decomposition is better than ZCA-based in Huang 2018. Specifically, could the authors explain why WC is better than W_{aca}C in Table 3?
- The authors claim progressive GAN used a larger generator to achieve a better performance than WC. The WC layer is generally larger than BN layer and has more learnable parameters. Could the authors compare  the number of parameter of generator in BN-ResNet, WC-ResNet, and progressive GAN?
- In Table 3, std-C is better than WC-diag, which indicates coloring is more important. In Table 6, cWC-diag is better than c-std-C, which indicates whitening is more important. Why?
- What is the batch size used for training? For conditional WC, do the samples in each minibatch have same label?
- Having ImageNet results will be a big support for the paper.
===========  comments after reading rebuttal ===========
I appreciate the authors' feedback. I raised my score for Fig 7 showing the conditional images, and for experiments on ImageNet.
I think WC is a reasonable extension to BN, and I generally like the extensive experiments. However, the paper is still borderline to me for the following concerns.
- I strongly encourage the authors to shorten the paper to the recommended 8-page.
- The motivation of WC for GAN is still unclear. WC is general extension of BN, and a simplified version has been shown to be effective for discrimination in Huang 2018. I understand the empirically good performance for GAN. But I am not convinced why WC is particularly effective for GAN, comparing to discrimination. The smoothness explanation of BN applies to both GAN and discrimination. I actually think it may be nontrivial to extend the smoothness argument from BN to WC.
- The motivation of cWC is still unclear. I did not find the details of cBN for class-label conditions, and how they motivated it in (Gulrajani et al. (2017) and (Miyato et al. 2018). Even if it has been used before, I would encourage the authors to restate the motivation in the paper. Saying it has been used before is an unsatisfactory answer for an unintuitive setting.
- Another less important comment is that it is still hard to say how much benefits we get from the more learnable parameters in WC than BN. It is probably not so important because it can be a good trade-off for state-of-the-art results. In table 3 for unconditioned generation, it looks like the benefits come a lot from the larger parameter space. For conditioned generation in table 6, I am not sure if whitening is conditioned or not, which makes it less reliable to me. If whitening is conditioned, then the samples in each minibatches may not be enough to get a stable whitening. If whitening is unconditioned, then there seems to be a mismatch between whitening and coloring.
====== second round after rebuttal =============
I raise the score again for the commitment of shortening the paper and the detailed response from the authors. That being said, I am not fully convinced about motivations for WC and cWC.
- GAN training is more difficult and unstable, but that does not explain why WC is particularly effective for GAN training.
- I have never seen papers saying cBN/cWC is better than other conditional generator conditioned on class labels. I think the capacity argument is interesting, but I am not sure if it applies to convolutional net (where the mean and variance of a channel is used), or how well it can explain the performance because neural nets are overparameterized in general. I would encourage authors to include these discussions in the paper.


Spoken language understanding (SLU) is a critical component in building dialogue systems. When building models for novel natural language domains, a major challenge is the lack of data in the new domains, no matter whether the data is annotated or not. Recognizing and annotating "intent" and "slot" of natural languages is a time-consuming process. Therefore, spoken language understanding in low resource domains remains a crucial problem to address. In this paper, we address this problem by proposing a transfer-learning method, whereby a SLU model is transferred to a novel but data-poor domain via a deep neural network framework. We also introduce meta-learning in our work to bridge the semantic relations between seen and unseen data, allowing new intents to be recognized and new slots to be filled with much lower new training effort. We show the performance improvement with extensive experimental results for spoken language understanding in low resource domains. We show that our method can also handle novel intent recognition and slot-filling tasks. Our methodology provides a feasible solution for alleviating data shortages in spoken language understanding.

Summary: The authors present a network which facilitates cross-domain
learning for SLU tasks where the the goal is to resolve intents and
slots given input utterances. At a high level, the authors argue that
by fine-tuning a pre-trained version of the network on a small set of
examples from a target-domain they can more effectively learn the
target domain than without transfer learning.
Feedback:
* An overall difficulty with the paper is that it is hard to
distinguish the authors' contributions from previous works. For
example, in Section 3.1, the authors take the model of Goyal et al. as
a starting point but explain only briefly one difference
(contatenating hidden layers). In Section 3.2 the contributions
becomes even harder to disentangle. For example, how does this section
relate to other word-embeddings papers cited in this section? Is the
proposed method a combination of previous works, and if not, what are
the core new ideas?
* Some sections are ad-hoc and should be justified/explained
better. For example, the objective, which ultimately determines the
trained model behaviour uses a product of experts formulation, yet the
authors do not discuss this. Similarly, the overarching message, that
by fine-tuning a suitable model initialisation using small amounts of
data from the target domain is fairly weak as the authors do not
detail exactly how the model is fine-tuned. Presumably, given only a
small number of examples, this fine-tuning runs the risk of
overfitting, unless some form of regularisation is applied, but this
is not discussed.
* Lastly, there are some curious dips in the plots (e.g., Figure 2 bottom left, Figure 3 top left, bottom left), which deserve more explanation. Additionally, the evaluation section could be improved if the scores were to show error-bars.
Minor: All plots should be modified so they are readable in grey-scale.


UNIVERSAL TRANSFORMERS
Recurrent neural networks (RNNs) sequentially process data by updating their state with each new data point, and have long been the de facto choice for sequence modeling tasks. However, their inherently sequential computation makes them slow to train. Feed-forward and convolutional architectures have recently been shown to achieve superior results on some sequence modeling tasks such as machine translation, with the added advantage that they concurrently process all inputs in the sequence, leading to easy parallelization and faster training times. Despite these successes, however, popular feed-forward sequence models like the Transformer fail to generalize in many simple tasks that recurrent models handle with ease, e.g. copying strings or even simple logical inference when the string or formula lengths exceed those observed at training time. We propose the Universal Transformer (UT), a parallel-in-time self-attentive recurrent sequence model which can be cast as a generalization of the Transformer model and which addresses these issues. UTs combine the parallelizability and global receptive field of feed-forward sequence models like the Transformer with the recurrent inductive bias of RNNs. We also add a dynamic per-position halting mechanism and find that it improves accuracy on several tasks. In contrast to the standard Transformer, under certain assumptions UTs can be shown to be Turing-complete. Our experiments show that UTs outperform standard Transformers on a wide range of algorithmic and language understanding tasks, including the challenging LAMBADA language modeling task where UTs achieve a new state of the art, and machine translation where UTs achieve a 0.9 BLEU improvement over Transformers on the WMT14 En-De dataset.

My summary: A new model, the UT, is based on the Transformer model, with added recurrence and dynamic halting of the recurrence. The UT should unite the computational universality properties of Neural Turing Machines and Neural GPU with good performance on disparate language and algorithmic tasks.
(I have read your author feedback and have modified my rating according to my understanding.)
Review:
The paper is well written and proofread, concrete and clear. The model is quite clearly explained, especially with the additional space of the supplementary material, appendices A and B (note fig 4 is less good quality than fig 2 for some reason) -- I’m fine with the use of the Supp Mat for this purpose.

The experiments have been conducted well, and demonstrate a wide range of tasks, which seems to suggest that the UT has pretty general purpose. The range of algorithmic tasks is limited, e.g. compared to the NTM paper.
I miss any experimental details at all on training.
I miss a comparison to Neural GPU and Stack RNN in 3.1, 3.2.
I miss a proof that the UT is computationally equivalent to a Turing machine. It does not have externally addressable, shared memory like a tape, and I’m not sure how to transpose read/write heads either.
The argument that the UT offers a good balance between inductive bias and expressivity is weak, though it may be the best one can hope for of a statistical model in a way. I note that in 3.1, the Transformer overfits, while it seems to underfit in 3.3 (lower LM and RC accuracy, higher LM perplexity), while the UT fare well, which suggests that the UT hits the balance better than the Transformer, at least.
From the point of view of network structure, it seems natural to lift further constraints on the model:
why should width of intermediate layers be exactly equal to sequence length?
why should all hidden state vectors be size $d$, the size of the embeddings chosen at the first layer, which might be chosen out of purely practical reasons like the availability of pre-trained word embeddings?
What is the contribution of this work? It starts from the Transformer, the ACT idea for dynamic halting in recurrent nets, the need for models fit for algorithmic tasks.
The UT’s building blocks are near-identical to the Transformers (and the paper is upfront and does a good job of explaining these similarities, fortunately)
- cf eq1-5: residuals, multi-headed self attention, and layer norm around all this.
- shared weights among all such units
- encoder-decoder architecture
- autoregressive decoder with teacher forcing
- decoder units like the encoder’s but with extra layer of attention to final output of encoder
- coordinate embeddings
The authors may correct me, but I believe that the UT with FC layers is exactly identical to the Transformer described in Vaswani 2017 for T=6.
So this paper introduces the idea of varying T, interprets it as a form of recurrence, and adds dynamic halting with ACT to that. Interestingly, the recurrence is not over sequence positions here.
This contribution is not major, on the other hand the experimental validation suggests the model is promising.
Typos and writing suggestions
above eq 8: masked such that -> masked so that
eq 8: dimensions of O and H^T are incompatible: d*V, m*d; to evacuate the notation issue for transposition, cf footnote 1, here and elsewhere, you could use either ${^t A}$ or $A^\top$ or $A^\intercal$. You could also write $t=T$ instead of just $T$.
sec3.3 line -1: designed such that -> designed so that
Towards the beginning of the paper, it may be useful to stabilise terminology for $t$: depth (as opposed to width for $m$), time steps, recurrence dimension, revisions, refinements


Implicit generative models are difficult to train as no explicit probability density functions are defined. Generative adversarial nets (GANs) propose a minimax framework to train such models, which suffer from mode collapse in practice due to the nature of the JS-divergence. In contrast, we propose a learning by teaching (LBT) framework to learn implicit models, which intrinsically avoid the mode collapse problem because of using the KL-divergence. In LBT, an auxiliary explicit model is introduced to learn the distribution defined by the implicit model while the later one's goal is to teach the explicit model to match the data distribution. LBT is formulated as a bilevel optimization problem, whose optimum implies that we obtain the maximum likelihood estimation of the implicit model. We adopt an unrolling approach to solve the challenging learning problem. Experimental results demonstrate the effectiveness of our method.

This work introduces a framework for learning implicit models that is robust to mode collapse. It consists in learning an explicit model of the implicit model through maximum likelihood while the later is used to teach the explicit model to better match the data distribution. The resulting bi-level optimization is carried out with truncated unrolled stochastic gradient descent.
# Quality
The method combines an interesting set of ideas. It is validated on some reasonable experiments.
However after reading the paper, I remain with too many unanswered questions:
- Why should the method avoid mode collapse? Experiments clearly show that it indeed is resilient to mode collapse, but I have would have been curious in seeing some more discussion regarding this point. What is the exact mechanism that solves the issue?
- What is the effect of K? Is mode collapse solved only because of the unrolled gradients?
- What is the effect of M? How does the method behave for M=1, as usually done in GANs?
- What if the explicit model has not enough capacity?
- The original Unrolled GAN paper presents better results for the ring problem. Why are results worse in the experiments?
More fundamentally what is the main benefit of this approach with respect to models that can be trained straight with maximum likelihood? (e.g., flow-based neural generative models; and as required for the explicit model) Is it only to produce generative models that are fast (because they are implicit)? Why not training only the explicit model directly on the data?
# Clarity
The paper is in general well-written, although some elements could be removed to actually help with the presentation.
- The development around influence functions could be removed, as the method ends up instead making use of truncated unrolled gradients.
- The theoretical analysis is straightforward and could be compressed in a single paragraph to motivate the method.
# Originality
The method makes use of several ideas that have been floating around and proposed in different papers. As far as I know, the combination proposed in this work is original.
# Significance
Results show clear resistance to mode collapse, which is an improvement for implicit models. However, other types of generative models generally do not suffer from this issue. Significance is therefore limited.


ADEF: AN ITERATIVE ALGORITHM TO CONSTRUCT ADVERSARIAL DEFORMATIONS
While deep neural networks have proven to be a powerful tool for many recognition and classification tasks, their stability properties are still not well understood. In the past, image classifiers have been shown to be vulnerable to so-called adversarial attacks, which are created by additively perturbing the correctly classified image. In this paper, we propose the ADef algorithm to construct a different kind of adversarial attack created by iteratively applying small deformations to the image, found through a gradient descent step. We demonstrate our results on MNIST with convolutional neural networks and on ImageNet with Inception-v3 and ResNet-101.

The paper proposes a new way to construct adversarial examples: do not change the intensity of the input image directly, but deform the image plane (i.e. compose the image with Id + tau where tau is a small amplitude vector field).
The paper originates from a document provably written in late 2017, which is before the deposit on arXiv of another article (by different authors, early 2018) which was later accepted to ICLR 2018 [Xiao and al.]. This remark is important in that it changes my rating of the paper (being more indulgent with papers proposing new ideas, as otherwise the novelty is rather low compared to [Xiao and al.]).
Pros:
- the paper is well written, very easy to read, well explained (and better formalized than [Xiao and al.]);
- the idea of deforming images is new (if we forget about [Xiao and al.]) and simple;
- experiments show what such a technique can achieve on MNIST and ImageNet. Interestingly, one can see on MNIST the parts of the numbers that the adversarial attack is trying to delete/create.
Cons:
- the paper is a bit weak, in that it is not very dense, and in that there is not much more content than the initial idea;
- for instance, more discussions about the results obtained could have been appreciated (such as my remark above about MNIST);
- for instance, a study of the impact of the regularization would have been interesting (how does the sigma of the Gaussian smoothing affect the type of adversarial attacks obtained and their performance -- is it possible to fool the network with [very] smooth deformations?);
- for instance, what about generating adversarial examples for which the network would be fully (wrongly) confident? (instead of just borderline unsure); etc.
- The interpolation scheme (how is defined the intensity I(x,y) for a non-integer location (x,y) within the image I) is rather important (linear interpolation, etc.) and should be at least mentioned in the main paper, and at best studied (it might impact the gradient descent path and the results);
- question: does the algorithm converge? could there be a proof of this? This is not obvious, as the objective potentially changes with time (selection of the current m best indices k of |F_k - F_l|). Also, the final overshoot factor (1+eta) is not very elegant, and not guaranteed to perform well if tau* starts being not small compared to the second derivative (i.e. g''.tau^2 not small) while I guess that for image intensities, spatial derivatives can be very high if no intensity smoothing scheme is used.
- note: the approximation tau* = sum_i tau_i (section 2.3) does not stand in the case of non-small deformations.
- still in section 2.3, I do not understand the statement "given that \nabla f is moderate": where does this property come from? or is "given" meant to be understood as "provided..." (i.e. under the assumption that...)?
- computational times could have been given (though I guess they are reasonable).
Other remarks:
- suggestion: I find the "slight abuse of notation" (of confusing the derivative with the gradient) a bit annoying and suggest to use a different symbol, such as \nabla g. This could be useful in particular in the following perspective:
- Mathematical side note: the "gradient" of a functional is not a uniquely-defined object in that it depends on the metric chosen in the tangent space. More clearly: the space of small deformations tau comes with an inner product (here L2, but one could choose another one), and the gradient \nabla g obtained depends on this inner product choice M, even though the derivative Dg is the same (they are related by Dg(tau) = < \nabla_M g | tau >_M for any tau). The choice of the metric can then be seen as a prior over desired gradient descent paths. In the paper, the deformation fields get smoothed by a Gaussian filter at some point (eq. 7), in order to be smoother: this can be interpreted as a prior (gradient descent paths should be made of smooth deformations) and as an associated inner product change (there do exist a metric M such that the gradient for that metric is \nabla_M g = S \nabla_L2 g). It is possible to favor other kind of deformations (not just smooth ones, but for instance rigid ones, etc. [and by the way this could make the link with "Manitest: Are classifiers really invariant?" by Fawzi and Frossard, BMVC 2015, who observe that a rigid motion can affect the classifier output]). If interested, you can check "Generalized Gradients: Priors on Minimization Flows" by Charpiat et al. for general inner products on deformations (in particular favoring rigid motion), and "Sobolev active contours" by Sundaramoorthi et al. for inner products more dedicated to smoothing (such as with the H1 norm).
- Note: about the remark in section 3.2: deformation-induced transformations are a subset of all possible transformations of the image (which are all representable with intensity changes), so it is expected that a training against attacks on the intensity performs better than a training against attacks on spatial deformations.


A belief persists long in machine learning that enlargement of margins over training data accounts for the resistance of models to overfitting by increasing the robustness. Yet Breiman shows a dilemma (Breiman, 1999) that a uniform improvement on margin distribution does not necessarily reduces generalization error. In this paper, we revisit Breiman's dilemma in deep neural networks with recently proposed normalized margins using Lipschitz constant bound by spectral norm products. With both simplified theory and extensive experiments, Breiman's dilemma is shown to rely on dynamics of normalized margin distributions, that reflects the trade-off between model expression power and data complexity. When the complexity of data is comparable to the model expression power in the sense that training and test data share similar phase transitions in normalized margin dynamics, Rademacher complexities of Lipschitz-normalized networks can be regarded as small constants and two efficient ways are derived via classic marginbased generalization bounds to successfully predict the trend of generalization error. On the other hand, over-expressed models that exhibit uniform improvements on training normalized margins may lose such a prediction power and fail to prevent the overfitting.

The authors found that general generalization bounds fail to capture the ramp loss. However, once the network scaled by its Lipschitz constant, it becomes efficient to get an upper bound of generalization error, while also needs to trade-off the constant in the margin error. Due to the limitation of fixing the constant in margin error, the authors  tried to use the quantile margin to change the bound, which is easy to tune the hyper-parameter. They also conducted the experiments that the quantile margin generalization bound could be used to predict the tendency of loss curve both in training and test in some sense.
It's really an interesting work to provide a way for early stopping and to show the quantile margin maybe a substitution of tendency in training error as well as test error.
Questions:
It could be difficult to judge from the phase transition, if exists, in the evolution of normalized margin distributions curve. Maybe  some quantitative descriptions are needed.
Besides, the authors' quantile margin bound (Theorem 2) shows the upper bound of margin (or say margin error). But the bound is not direct to support the powerful experiments results, the relationship between the tendency of quantile margin, training and test error.
Typos:
 In Eqn. (10), the first $f_t$ should be $\widetilde{f}_t$ .
In Eqn. (9) and (11), there is  $1$.
In Proof in Lemma A.1, the convolution operator is $x(v)$ not $x(u)$, since Lemma is also true.
In Proof in Lemma D.4, though the proof is same in the book `Foundations of machine learning' by Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar, please check the typo.
In Proof of Proposition 1, lack $\frac{1}{n}$ in Rademacher complexity.
In Proof of Theorem 1, maybe you should take $\mathbb{E}$ not $\mathbb{P}$ before $\ell_{\gamma_1,\gamma_2}(\xi(\widetilde{f}(x,y))))$.
In Proof of Theorem 2, page 18 the last line in the equation, why can the second term after divided by $L_f$ bounded by $L$, maybe need some conditions or I missed something.


Many applications in machine learning require optimizing a function whose true gradient is unknown, but where surrogate gradient information (directions that may be correlated with, but not necessarily identical to, the true gradient) is available instead. This arises when an approximate gradient is easier to compute than the full gradient (e.g. in meta-learning or unrolled optimization), or when a true gradient is intractable and is replaced with a surrogate (e.g. in certain reinforcement learning applications or training networks with discrete variables). We propose Guided Evolutionary Strategies, a method for optimally using surrogate gradient directions along with random search. We define a search distribution for evolutionary strategies that is elongated along a subspace spanned by the surrogate gradients. This allows us to estimate a descent direction which can then be passed to a first-order optimizer. We analytically and numerically characterize the tradeoffs that result from tuning how strongly the search distribution is stretched along the guiding subspace, and use this to derive a setting of the hyperparameters that works well across problems. Finally, we apply our method to example problems including truncated unrolled optimization and training neural networks with discrete variables, demonstrating improvement over both standard evolutionary strategies and first-order methods (that directly follow the surrogate gradient). We provide a demo of Guided ES at: redacted URL

Summary: The paper proposes a method to improve random search by building a subspace of the previous k surrogate gradients, mixing it with an isotropic Gaussian distribution to improve the search. Results reported shows are good compared to other approaches for learning weights of neural networks. However, paper lacks clarity and is quite hard to follow.
Quality: The paper presents a well-designed approach that is able to deal with optimization in high dimensionality space, by building a lower order surrogate model build upon the previous gradients computed with this surrogate model. The analysis appears to be correct and provide credential to the approach. Results reported are very good. However, testing is relatively limited to few cases. More experimental results on a good set of problems with several methods would have made the paper stronger and more convincing.
Clarity: The paper is hard to follow. Maybe because I am not completely familiar with the topic, but many elements presented lacks some context. The authors appear clearly to be knowledgeable of their topics, but lacks the capacity to provide all required background to follow their thoughts. The method could have been better illustrated, I found that Fig. 1a not enough to explain the method, while Fig. 1b and other training curves not useful to understand the approach. Some pseudo-code to illustrate the use of the proposed method might certainly help to improve clarity. Sec. 3.2 is not enough to understand well the approach.
Originality: The approach is allowing a nice trade-off between pure random search and guide search through a surrogate model over a subspace of limited dimensionality. This is in-line of some work on the use of ES for training neural networks, but I am not aware of other similar work although I am not super knowledgeable of the field.
Significance: The approach can have its impact for optimizing deep networks with no gradient, but more exhaustive experimental testing would be required.
Pros and cons:
+ Sound approach
+ Good theoretical support of the approach (bias-variance analysis)
+ Great results reported
+ Of importance for optimizing without gradients
- Presentation of the method lacking many details and not very clear
- Overall quality of the paper is subpar, tend to be very textual and hard to follow in several parts
- Experiments are not exhaustive and detailed. Loss plots are provided for some methods compared. Looks more like a preliminary validation.
I think that if the paper can be rewritten to be more tight, clearer in its presentation, with figures and pseudo-code to illustrate the method better, with more exhaustive testing, it can be really great. Current, the method appears to be great, but the writing quality of the paper is not yet there.


The linear and non-flexible nature of deep convolutional models makes them vulnerable to carefully crafted adversarial perturbations. To tackle this problem, in this paper, we propose a nonlinear radial basis convolutional feature transformation by learning the Mahalanobis distance function that maps the input convolutional features from the same class into tight clusters. In such a space, the clusters become compact and well-separated, which prevent small adversarial perturbations from forcing a sample to cross the decision boundary. We test the proposed method on three publicly available image classification and segmentation data-sets namely, MNIST, ISBI ISIC skin lesion, and NIH ChestX-ray14. We evaluate the robustness of our method to different gradient (targeted and untargeted) and nongradient based attacks and compare it to several non-gradient masking defense strategies. Our results demonstrate that the proposed method can boost the performance of deep convolutional neural networks against adversarial perturbations without accuracy drop on clean data.

The authors propose a new defense against adversarial examples based on radial basis features. Prior work has suggested that the linearity of standard convolutional networks may be a factor contributing to their vulnerability against adversarial examples, and that radial basis functions may help alleviate this weakness. The current paper builds on this idea and proposes a concrete way to add radial basis features to existing convnet architectures.
In the proposed approach, each layer of the network is augmented with a radial basis feature transform of the features in this layer. The output of this feature transform is then concatenated with the features in this layer. The centers of the radial basis features, the bandwidth, and the distance matrix are trained with the other network parameters. The distance matrix in the feature transform is used to compute the Mahalanobis distance between the features and centers in the radial basis functions.
The authors evaluate their defense experimentally on the standard MNIST dataset and two medical image datasets: X-chest14 for classification and a 2D RGB skin lesion dataset from the 2017 IEEE ISBI International Skin Imaging Collaboration (ISIC) Challenge for image segmentation. The experiments show that their method improves over an undefended network on MNIST. On X-chest14, their method improves over features squeezing (input quantization) and Gaussian data augmentation. On the image segmentation dataset, the method improves over these baselines as well as adversarial training.
While I find the overall idea interesting, I have some doubts about the experimental evaluation. For instance, the authors do not compare their MNIST numbers to the robust optimization results reported in Madry et al. (cited in the paper). Robust optimization achieves higher adversarial accuracy than the numbers reported in Table 1.
More importantly, it is unclear to what extent unmodified first-order methods are effective for the proposed defense. While the authors investigate whether their networks exhibit gradient masking / obfuscation, the left plot in Figure 3 still leaves some questions. Based on the curves for FGSM and BIM, the proposed defense would still achieve a high accuracy even against attacks with eps = 0.5. However, this would be a clear failure of the first order attacks (but not a sign of true robustness) because an adversary with eps = 0.5 can trivially defend any network by setting every input pixel to 0.5. Hence the authors should investigate what happens in the regime between eps = 0.4 and eps = 0.5.
While I support the use of non-standard datasets for evaluation, it would still strengthen the paper if the author also reported accuracy numbers on CIFAR-10. The X-chest14 and the segmentation dataset have not been frequently used in the adversarial robustness literature to the best of my knowledge. Hence it is less clear how well the proposed methods perform on these datasets.
While I find the overall idea interesting, with the current experimental evaluation I unfortunately cannot recommend accepting the paper.
Further comments:
- The distinction between "data-level" and "algorithmic-level" approaches in the introduction is unclear to me. Adversarial training can also be seen as robust optimization, which is arguably an algorithmic approach.
- At the beginning of Section 2, it would be helpful if the authors first introduced the meaning of the variables n, m, and k before using them. In general, it would be helpful if the authors described in more detail how the radial basis features are incorporated into the network.
- How is the adversarial training baseline in Section 4.2 implemented? The choice of adversary in adversarial training / robust optimization can be crucial for the robustness of the resulting model.
- Since the authors refer to the linearity of existing model as a potential weakness: there are also alternative explanations, e.g., see https://arxiv.org/abs/1801.02774 and https://arxiv.org/abs/1804.11285 .
- The test sets used in the evaluation are fairly small (150 and 200 data points). In this regime, 95% confidence intervals can be as large as +/- 8%. Hence I would recommend increasing the size of the test sets to at least 1,000.


Deep neural networks have been demonstrated to be vulnerable to adversarial attacks, where small perturbations intentionally added to the original inputs can fool the classifier. In this paper, we propose a defense method, Featurized Bidirectional Generative Adversarial Networks (FBGAN), to extract the semantic features of the input and filter the non-semantic perturbation. FBGAN is pre-trained on the clean dataset in an unsupervised manner, adversarially learning a bidirectional mapping between the high-dimensional data space and the low-dimensional semantic space; also mutual information is applied to disentangle the semantically meaningful features. After the bidirectional mapping, the adversarial data can be reconstructed to denoised data, which could be fed into any pre-trained classifier. We empirically show the quality of reconstruction images and the effectiveness of defense.

This work proposes to defend against adversarial examples by “denoising” the input image through an autoencoder (a BiGAN trained similar to InfoGAN) before classifying it with a standard CNN. The robustness of the model is evaluated on the L_infinity metric against FGSM and PGD.
My main criticism is as follows:
* Novelty: several defences are based on a similar principle and the contributions of this paper are unclear.
* Insufficient evidence: The evaluation is minimal (only FGSM and PGD, no decision-, transfer- or score-based attacks) and insufficient to support the claims.
* Gradient masking: There is at least one clear sign of gradient masking in the results (FGSM performing better than PGD).
### Novelty
The only prior work against which the paper compares is DefenseGAN. The only advantage over DefenseGAN being stated is performance (because no intermediate optimisation step is used). However, besides DefenseGAN there are several other defences that project the input onto the learned manifold of “natural” inputs, including (see prior work section in [1] for an up-to-date list):
* Adversarial Perturbation Elimination GAN
* Robust Manifold Defense
* PixelDefend (autoregressive probabilistic model)
* MagNets
### Insufficient evidence
The only attacks employed are two gradient-based techniques (FGSM and PGD). It is known that gradient-based techniques may suffer from gradient-masking (see also next point) and that the effectiveness of different attacks various greatly (which is why one should use many different attacks). Hence, a full evaluation of the model should include score-based and decision-based attacks.
### Gradient masking
In Figure 5 (b) the FGSM attack performs better than PGD for epsilon = 0.05 (66.4% vs 71.5%). PGD, however, should be strictly more powerful than FGSM if the gradients and the hyperparameters are ok.
Gradient masking is the primary reason for why 95% of all proposed defences turned out to be ineffective, and there are good reasons to believe that the same might affect this defence. The robustness evaluation has to be much more thorough and convincing before any substantiated claims about the bidirectional architecture proposed here can be derived. In addition, the difference to prior work has to be made much clearer.
[1] Schott et al. “Towards the first adversarially robust neural network model on MNIST”


Recently, image-to-image translation has seen a significant success. Among many approaches, image translation based on an exemplar image, which contains the target style information, has been popular, owing to its capability to handle multimodality as well as its suitability for practical use. However, most of the existing methods extract the style information from an entire exemplar and apply it to the entire input image, which introduces excessive image translation in irrelevant image regions. In response, this paper proposes a novel approach that jointly extracts out the local masks of the input image and the exemplar as targeted regions to be involved for image translation. In particular, the main novelty of our model lies in (1) co-segmentation networks for local mask generation and (2) the local mask-based highway adaptive instance normalization technique. We demonstrate the quantitative and the qualitative evaluation results to show the advantages of our proposed approach. Finally, the code is available at https://github.com/AnonymousIclrAuthor/ Highway-Adaptive-Instance-Normalization.

Summary--
The paper tries to address an issue existing in current image-to-image translation at the point that different regions of the image should be treated differently. In other word, background should not be transferred while only foreground of interest should be transferred. The paper propose to use co-segmentation to find the common areas to for image translation. It reports the proposed method works through experiments.
There are several major concerns to be addressed before considering to publish.
1) The paper says that "For example, in a person’s facial image translation, if the exemplar image has two attributes, (1) a smiling expression and (2) a blonde hair, then both attributes have to be transferred with no other options", but the model in the paper seems still incapable of transferring only one attribute. Perhaps an interactive transfer make more sense, while co-segmentation does not distinguish the part of interest to the user. Or training a semantic segmentation make more sense as the semantic segment can specify which region to transfer.
2) As co-segmentation is proposed to "capture the regions of a common object existing in multiple input images", why does the co-segmentation network only capture the eye and mouth part in Figure 2 and 3, why does it capture the mouth of different shape and style in the third macro column in Figure 4 instead of eyes? How to train the co-segmentation module, what is the objective function? Why not using a semantic segmentation model?
3) The "domain-invariant content code" and the "style code" seem rather subjective. Are there any principles to design content and style codes? In the experiments, it seems the paper considers five styles to transfer as shown in Table 1. Is the model easy to extend to novel styles for image translation?
4) What does the pink color mean in the very bottom-left or top-right heatmap images in Figure 2? There is no pink color reference in the colorbar.
5) Figure 5: Why there is similariy dark patterns on the mouth? Is it some manual manipulation for interactive transfer?
6) Though it is always good to see the authors are willing to release code and models, it appears uncomfortable that github page noted in the abstract reveals the author information. Moreover, in the github page,
even though it says "an example is example.ipynb", the only ipynb file contains nothing informative and this makes reviewers feel cheated.
Minor--
There are several typos, e.g., lightinig.


We introduce Dynamic Planning Networks (DPN), a novel architecture for deep reinforcement learning, that combines model-based and model-free aspects for online planning. Our architecture learns to dynamically construct plans using a learned state-transition model by selecting and traversing between simulated states and actions to maximize valuable information before acting. In contrast to modelfree methods, model-based planning lets the agent efficiently test action hypotheses without performing costly trial-and-error in the environment. DPN learns to efficiently form plans by expanding a single action-conditional state transition at a time instead of exhaustively evaluating each action, reducing the required number of state-transitions during planning by up to 96%. We observe various emergent planning patterns used to solve environments, including classical search methods such as breadth-first and depth-first search. DPN shows improved data efficiency, performance, and generalization to new and unseen domains in comparison to several baselines.

This paper proposes a new architecture for model-based deep RL, in which an “inner agent” (IA) takes several planning steps to inform an “outer agent” (OA) which actually acts in the world. The main contributions are to propose a new objective for the IA, and to allow the IA to “undo” its imagined actions. Overall I think this could be a great paper, but it needs further justification of some of the architectural choices and more rigorous analysis/experiments before it will be ready for acceptance.
Pros:
- Nice demonstration of improved data efficiency over existing model-based methods.
- Substantial improvement over other model-based methods in terms of computational cost.
- Interesting qualitative analysis showing discovery of DFS and BFS-like search procedures.
Cons:
- Limited novelty over existing methods.
- It is unclear what in the model contributes to improved performance.
Quality
---------
The results in the paper seem impressive in terms of sample complexity, but I think there needs to be further exploration of the source of the results. I strongly suggest including in a revision a number of ablation experiments to tease these details apart---for example, what do the results look like if the IA uses the same objective as the OA? Does the agent achieve worse performance if it has to restart its imaginations from the root of the tree each time, as is more analogous to MCTS and other previous model-based approaches?
Additionally, there are a few places in the paper where unjustified statements are made. For example, in Section 5.3, the paper states that “we hypothesize that focusing, by repeatedly visiting the same state, the IA ensures that the POI is remembered in its hidden state such that the OA can act accordingly, given this information”. This seems very speculative. It would certainly be very interesting if true, but there needs to be something more than just intuition to back up this hypothesis. I recommend including some probe experiments (e.g., force the IA to take a sequence of such actions, or not, and see what the result on the behavior of the OA is) or removing speculations such as this (or moving it to the appendix).
The literature review is missing some related work, particularly from the realm of model-based continuous control. [1-3] are a few references to start with; these papers take a different approach in that they don’t use tree search but they are still worthwhile discussing. I think a reference to [4] is also missing, which takes a related approach to learning the decisions needed to perform MCTS.
Clarity
--------
Overall, the paper is well-written and I understand what was done and how the architecture works. However, I had a hard time understanding the choice of the inner objective (Equation 1). The paper states that this equation defines the “value of information” and defines it as the product of the KL from the OA’s hidden state prior to the transition to after the transition, multiplied by the Q value estimated by the OA and the action probability of the IA. This is very mysterious to me. Why is this a good objective? Why does the KL term of the hidden state of the OA have anything to do with the value of information? Given that the difference in objective of the IA is one of the main contributions of the paper, this choice needs to be justified, explained, and examined. As mentioned above, it would be best if a revision could include some ablation experiments where the choice of this objective is more closely examined.
More broadly, as mentioned above, it is unclear to me what part of the framework results in improved performance. Is it the ability to “undo” actions (rather than starting over from the root or exhaustively performing BFS), or is it the KL-based reward given to the IA? The paper does not provide any insight into this question, making it unclear what are the key points I should take away.
Minor:
- The colors in the caption of Figure 5 do not match the colors in the figure.
- The colors in Figure 7 are very dark and it is hard to make out what is actually happening in the figure.
Originality
-------------
The objective of the inner agent (Equation 1) appears novel, though as discussed above it is unclear to me what exactly it means and what its implications are. The idea of constructing an imagination tree state-by-state is not particularly novel, and was previously explored by Pascanu et al. (2017). I think this paper deserves more discussion and comparison than it is given in the present work (in particular, compare Figure 3 of the present paper and Figure 2 of Pascanu et al.). In general, the main idea in both papers is the same: have an agent learn to take internal planning steps and construct a planning tree that then informs the final action in the world. The biggest differences from Pascanu et al. are that the present work uses a separate objective for the inner agent, and allows taking a step backwards and returning to the previous state (whereas Pascanu et al. only allowed imagining from the current imagined state or from the root). So, the overall the paper has some new ideas, but is not highly novel compared to previous work. I see the two biggest original contributions as being: (1) the separate objective in the inner agent and (2) the ability for the agent to restart its imagination from the previous imagined state.
Significance
----------------
The results reported by the paper are significant in that they do show dramatic improvement in sample complexity over existing model-free methods, as well as improvement in computational cost over existing model-based methods. However, as discussed above, it is hard for me to know what conclusions I should draw from the paper in terms of what aspects of the approach drive this performance. Thus, I think the lack of clarity in this respect limits the significance of the paper.
[1] Finn, C., & Levine, S. (2017). Deep visual foresight for planning robot motion. In Proceedings of the International Conference on Robotics and Automation (ICRA 2017).
[2] Srinivas, A., Jabri, A., Abbeel, P., Levine, S., & Finn, C. (2018). Universal planning networks. In Proceedings of the 35th International Conference on Machine Learning (ICML 2018).
[3] Henaff, M., Whitney, W., & LeCun, Y. (2018). Model-based planning with discrete and continuous actions. arXiv preprint arXiv:1705.07177
[4] Guez, A., Weber, T., Antonoglou, I., Simonyan, K., Vinyals, O., Wierstra, D., … Silver, D. (2018). Learning to search with MCTSnets. In Proceedings of the 35th International Conference on Machine Learning (ICML 2018).


Optimal Transport offers an alternative to maximum likelihood for learning generative autoencoding models. We show how this principle dictates the minimization of the Wasserstein distance between the encoder aggregated posterior and the prior, plus a reconstruction error. We prove that in the non-parametric limit the autoencoder generates the data distribution if and only if the two distributions match exactly, and that the optimum can be obtained by deterministic autoencoders. We then introduce the Sinkhorn AutoEncoder (SAE), which casts the problem into Optimal Transport on the latent space. The resulting Wasserstein distance is minimized by backpropagating through the Sinkhorn algorithm. SAE models the aggregated posterior as an implicit distribution and therefore does not need a reparameterization trick for gradients estimation. Moreover, it requires virtually no adaptation to different prior distributions. We demonstrate its flexibility by considering models with hyperspherical and Dirichlet priors, as well as a simple case of probabilistic programming. SAE matches or outperforms other autoencoding models in visual quality and FID scores.

The paper introduces a new cost function for training Wasserstein Autoencoders that combines reconstruction error with Sinkhorn distance on the latent space. Authors provide nice theoretical motivation, yet empirical results seem incremental and do not fully support the effectiveness of this approach.
Pros:
- Theorem 3.1 (although trivial) provides motivation for optimizing Wasserstein distance in the latent space in WAEs.
- Theorem 3.2 shows sufficiency of optimization over deterministic encoders in WAEs.
- The proposed SAE virtually does not favor any prior and can preserve some aspects of geometry of the original space.
Cons:
- It is unclear why Sinkhorn algorithm would provide better estimate of Wasserstein distance than e.g. adversarial WGANGP (which would be a variant of GAN-WAE). Sinkhorn convergence is discussed only in terms of sample size and  smoothing regularizer, not in the context of batch training.
- Quantitative results are on par or marginally better than other methods, they also lack some comparisons (see details below).
- There is no comparison to relevant models outside VAE scope, e.g. ALI [4].
The novelty of this paper is combining WAEs with Sinkhorn algorithm. Overall, it has potential, but the proposed method would probably require clearer evaluation.
Detailed issues:
- Notation for posterior seems somewhat inconsistent and misleading, namely push-forward G#P_Z = P_G, while Q#P_X = Q_Z.
- It is unclear why MMD or GAN losses on WAS's latent space are referred to as heuristics, each of these constitutes a divergence in the same way as the proposed Sinkhorn distance.
- FID scores for MNIST are incomparable due to the use of own network; using LeNet has been proposed [3].
- It is unclear what ‘Empirical lower bounds’ for MMD mentioned in Table 1. caption mean, as unbiased MMD estimator (e.g. [2]) is available. On the other hand, FID is known to be biased [3], so test-set FID should be provided for comparison.
- Table 2. lacks comparison of SAE with normal prior even though a) authors note that MMDs are incomparable with different priors, b) SAEs is claimed to be prior-agnostic, c) in such setting MMD-WAE might be advantageous [1]. Again, no test-set FID scores.
- Samples in Figure 2 too small.
- MMD lacks citation (e.g. [2]).
Typos:
p.6 line 3 construcetion -> construction
p.6 line 30 Hypersherical -> Hyperspherical
P.8 line 1 this a sign -> this is a sign
[1] Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schölkopf. Wasserstein Auto-Encoders. ICLR 2018.
[2] Arthur Gretton, Karsten M. Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alex J. Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13, 2012a.
[3] Mikołaj Bińkowski, Dougal J. Sutherland, Michael Arbel, Arthur Gretton. Demystifying MMD GANs. ICLR 2018.
[4] Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky and Aaron Courville. Adversarially Learned Inference. ICLR 2017


INFORMATION ASYMMETRY IN KL-REGULARIZED RL
Many real world tasks exhibit rich structure that is repeated across different parts of the state space or in time. In this work we study the possibility of leveraging such repeated structure to speed up and regularize learning. We start from the KL regularized expected reward objective which introduces an additional component, a default policy. Instead of relying on a fixed default policy, we learn it from data. But crucially, we restrict the amount of information the default policy receives, forcing it to learn reusable behaviours that help the policy learn faster. We formalize this strategy and discuss connections to information bottleneck approaches and to the variational EM algorithm. We present empirical results in both discrete and continuous action domains and demonstrate that, for certain tasks, learning a default policy alongside the policy can significantly speed up and improve learning.

-- Originality --
This paper studies how to use KL-regularization with information asymmetry to speed up and improve reinforcement learning (RL). Compared with existing work, the major novelty in the proposed algorithm is that it uses a default policy learned from data, rather than a fixed default policy. Moreover, the proposed algorithm also limits the amount of information the default policy receives, i.e., there is an "information asymmetry" between the agent policy and the default policy. In many applications, the default policy is purposely chosen to be "goal agnostic" and hence conducts the "transfer learning". To the best of my knowledge, this "informationally asymmetric" KL-regularization approach is novel.
-- Clarify --
The paper is well written in general and is easy to follow.
-- Significance --
I think the idea of regularizing RL via an informationally asymmetric default policy is interesting. It might be an efficient way to do transfer learning (generalization) in some RL applications. This paper has also done extensive and rigorous experiments. Some experiment results are thought-provoking.
-- Pros and Cons
Pros:
1)  The idea of regularizing RL via an informationally asymmetric default policy is interesting. To the best of my knowledge, this "informationally asymmetric" KL-regularization approach is novel.
2) The experiment results are extensive, rigorous, and thought-provoking.
Cons:
1) My understanding is that this "informationally asymmetric" KL-regularization approach is a general approach and can be combined with many policy learning algorithms. It is not completely clear to me why the authors choose to combine it with an actor-critic approach (see Algorithm 1)? Why not combine it with other policy learning algorithms? Please explain.
2) This paper does not have any theoretical results. I fully understand that it is highly non-trivial or even impossible to analyze the proposed algorithm in the general case. However, I recommend the authors to analyze (possibly a variant of) the proposed algorithm in a simplified setting (e.g. the network has only one layer, or even is linear) to further strengthen the results.
3) The experiment results of this paper are interesting, but I think the authors can do a better job of intuitively explaining the experiment results. For instance, the experiment results show that when the reward is "dense shaping", the proposed method and the baseline perform similarly. Might the authors provide an intuitive explanation for this observation? I recommend the authors to try to provide intuitive explanation for all such interesting observations in the paper.