Constructing Unrestricted Adversarial Examples with Generative Models
Adversarial examples are typically constructed by perturbing an existing data point within a small matrix norm, and current defense methods are focused on guarding against this type of attack. In this paper, we propose unrestricted adversarial examples, a new threat model where the attackers are not restricted to small normbounded perturbations. Different from perturbation-based attacks, we propose to synthesize unrestricted adversarial examples entirely from scratch using conditional generative models. Specifically, we first train an Auxiliary Classifier Generative Adversarial Network (AC-GAN) to model the class-conditional distribution over data samples. Then, conditioned on a desired class, we search over the AC-GAN latent space to find images that are likely under the generative model and are misclassified by a target classifier. We demonstrate through human evaluation that unrestricted adversarial examples generated this way are legitimate and belong to the desired class. Our empirical results on the MNIST, SVHN, and CelebA datasets show that unrestricted adversarial examples can bypass strong adversarial training and certified defense methods designed for traditional adversarial attacks.

****** EDIT: I am upgrading my review score in response to the authors' rebuttal. However, the similarities with Song et al should be made much more explicit, and the reviewers' other comments should also be taken into account. *******  This paper proposes a technique for generating adversarial examples that are not slight perturbations of an existing data sample. Rather, they are samples that are designed to look like a certain class to an observer, but get classified to a pre-specified, different class. The approach for doing so is to define a loss function that combines these two objectives (while also enforcing that the latent code vector should be near a randomly-selected initial vector). The authors show that this method is able to generate successful adversarial examples against schemes that are designed to protect against perturbation-based defenses.   Overall, the paper seems to capture a more reasonable definition of adversarial examples; there is no inherent reason why adversarial examples should necessarily be based on an existing sample, and using GANs is an interesting way to decouple the adversarial example generation from the samples available in a dataset. Besides this insight, I didn’t feel like the ideas are particularly novel from a technical standpoint—the machine learning aspects of the paper are fairly standard. I think this work may be more interesting from a security standpoint than from a machine learning one. However, the authors did a fairly good job of evaluating their ideas (especially considering how subjective the problem is), and the paper was overall an enjoyable read.   The paper states that using a larger epsilon is not effective because the samples are obviously tampered with. At least visually, I thought the samples in Figure 1 also looked tampered with (or at least clearly distorted). The experiment seems a little strange to me, because the samples with larger epsilon mainly appear to have a different background in Figure 2. This is a very easy thing to detect, but it doesn’t mean that the image looks substantially different from a hand-written digit. In fact, to me, the images just look like MNIST digits with a fuzzed background. Perhaps this experiment would have been more informative on a different dataset, like CIFAR? My concern is that because MNIST is so simple, it is easy to see these distortions in background, whereas they might not be as visible on a richer background.   The paper is very well-written, though there are several typos that should be proofread.


End-to-end Symmetry Preserving Inter-atomic Potential Energy Model for Finite and Extended Systems
Machine learning models are changing the paradigm of molecular modeling, which is a fundamental tool for material science, chemistry, and computational biology. Of particular interest is the inter-atomic potential energy surface (PES). Here we develop Deep Potential -Smooth Edition (DeepPot-SE), an end-to-end machine learning-based PES model, which is able to efficiently represent the PES of a wide variety of systems with the accuracy of ab initio quantum mechanics models. By construction, DeepPot-SE is extensive and continuously differentiable, scales linearly with system size, and preserves all the natural symmetries of the system. Further, we show that DeepPot-SE describes finite and extended systems including organic molecules, metals, semiconductors, and insulators with high fidelity.

The paper proposes a new learnable model DeepPot-SE for inter-atomic potential energy surfaces (PES) based on deep neural networks. The authors start by introducing a number of requirements that a PES model should fulfil. Compared to other proposed model, this proposed model is the first to fulfil all these requirements, including differentiability and preserving natural symmetries. In the empirical evaluation, the performance of the proposed model is comparable to or better than state-of-the-art models as measured in MAE for energy and force predictions.  Generally the paper the paper is well written and easy to follow. However, the paper may be hard to follow for someone with little experience in PES modelling, e.g., it is never explained in the paper what is meant by "Finite and Extended Systems" (which is part of the title of the paper).  One of the central aspects of the paper is the feature matrix D^i in equation (11) that is invariant to translations, rotations and permutations. The authors have shown that some of the subexpressions of (11) are indeed invariant. However, given that this feature matrix plays such a central role in the paper, I think that the authors should give some details beyond "It is straightforward to see that D^i, and hence the DeepPot-SE model, preserve all the necessary symmetries" (line 141-142), e.g. in supplementary material.  The work seems to be an elegant improvement of the Deep Potential model [17], which unfortunately also makes the work seems a little incremental. However, the idea of the feature matrix D seems novel and could have potential applications outside the domain of PES modelling (as also noted by the authors). Such applications are however not considered.  The authors evaluate the model empirically on a wide selection of systems. For small organic molecules, the proposed DeepPot-SE model is compared to three state-of-the-art models. However, given that DeepPot-SE is an improvement of the Deep Potential model, I find it surprising that the Deep Potential model is not included as a baseline. Generally, the DeepPot-SE model has a performance that is comparable to the other models. However, it does not outperform the other models. The results for bulk systems are compared to the DeePMD model, and on these datasets, the proposed consistently outperforms the baseline model. However, for this dataset I am slightly puzzled about the training and test split: 90% of the snapshots of an MD trajectory are randomly selected as training data and 10% are selected as test data. As snapshots are only 10fs apart, I worry that the examples in the training and test set are highly similar. Could the authors please clarify this in their rebuttal.  To summarise, the paper is of high quality and clarity. The work has some both incremental aspects and some novel/original aspect (that may have application beyond PES). The results on small organic molecules are not highly significant, however, on bulk systems, the baseline model was systematically outperformed.   Minor comments:  1. Figure 1: ".= (b3)" -> ". (b3)"  2. Equation (8): it would be useful to remind the reader that r_ji = $|\vec{r}ji}|$.  3. Figure 2:  "DeepMD-SE" -> "DeepPot-SE"   Comments after rebuttal:   The authors have addressed most of my issue in their rebuttal, and I recommend accepting the paper. However, I would like to stress the importance of added a rigorous proof for that D^i (equation 11) preserves all the intended symmetries in the camera-ready version.


Information-theoretic Limits for Community Detection in Network Models
We analyze the information-theoretic limits for the recovery of node labels in several network models. This includes the Stochastic Block Model, the Exponential Random Graph Model, the Latent Space Model, the Directed Preferential Attachment Model, and the Directed Small-world Model. For the Stochastic Block Model, the non-recoverability condition depends on the probabilities of having edges inside a community, and between different communities. For the Latent Space Model, the non-recoverability condition depends on the dimension of the latent space, and how far and spread are the communities in the latent space. For the Directed Preferential Attachment Model and the Directed Small-world Model, the non-recoverability condition depends on the ratio between homophily and neighborhood size. We also consider dynamic versions of the Stochastic Block Model and the Latent Space Model.

This article proves information theoretic lower bounds for the community detection problem in a range of network models. For the SBM, this has been previously achieved in works of Mossel-Neeman-Sly, Coja-Oghlan, Abbe-Sandon etc. The authors of the current paper plant community structure (two randomly assigned communities) in various other models such as latent space models, preferential attachment models, small-world models etc. and prove similar information theoretic lower bounds. The proofs employ  (not surprisingly) Fano's inequality in carefully restricted submodels.   However, the authors only prove lower bounds, it is not clear if these are tight (it is for the SBM), and if so what (preferably polynomial time) algorithms can achieve them. There are a host of other interesting questions to ask. For example, is there an information-computation gap for larger number of communities as in the case of the SBM?  I found the paper interesting and believe that it is a good contribution to the community detection literature.  Here are some comments/suggestions for the authors:  1. Repetition of the sentences "we are interested in information theoretic limits. Thus, we define the Markov Chain Y* --> A --> Y. Using Fano's inequality..." before every theorem is not necessary. Say it once in the beginning perhaps. Similarly, the sentence "The use of restricted ensembles..." is repeated in page 4, line 138. 2. Page 1, line 33: "several social networks" should be "several social network models". 3. Page 3, line 98: "In a" should be "In an". 4. Page 7, line 230: "classic Small-world" should be "classic small-world phenomenon". 5. Add commas after the if clauses in the theorems (at end of the displayed equations). 6. Since your particular ERGM is essentially an SBM with different parametrization, results of Mossel-Neeman-Sly should apply. So, you should add that the corresponding lower bound is tight for this subclass. 7. Since you have more than a page left within the NIPS page limit, it would be nice if you could say something about algorithms that could be used for community extraction in these models and/or what has been done in the literature and in what parameter regimes they are successful.


Leveraged volume sampling for linear regression
Suppose an n ⇥ d design matrix in a linear regression problem is given, but the response for each point is hidden unless explicitly requested. The goal is to sample only a small number k ⌧ n of the responses, and then produce a weight vector whose sum of squares loss over all points is at most 1 + ✏ times the minimum. When k is very small (e.g., k = d), jointly sampling diverse subsets of points is crucial. One such method called volume sampling has a unique and desirable property that the weight vector it produces is an unbiased estimate of the optimum. It is therefore natural to ask if this method offers the optimal unbiased estimate in terms of the number of responses k needed to achieve a 1 + ✏ loss approximation. Surprisingly we show that volume sampling can have poor behavior when we require a very accurate approximation -indeed worse than some i.i.d. sampling techniques whose estimates are biased, such as leverage score sampling. We then develop a new rescaled variant of volume sampling that produces an unbiased estimate which avoids this bad behavior and has at least as good a tail bound as leverage score sampling: sample size k = O(d log d + d/✏) suffices to guarantee total loss at most 1 + ✏ times the minimum with high probability. Thus we improve on the best previously known sample size for an unbiased estimator, k = O(d 2 /✏). Our rescaling procedure leads to a new efficient algorithm for volume sampling which is based on a determinantal rejection sampling technique with potentially broader applications to determinantal point processes. Other contributions include introducing the combinatorics needed for rescaled volume sampling and developing tail bounds for sums of dependent random matrices which arise in the process.

This paper studies deficiencies of volume sampling, and proposes a modification based on leverage scores, or renormalizing the current ellipse before performing volumne rejection sampling. It improves the number of unbiased samples required to guarantee 1\pm\epsilon accuracy by a factor of \epsilon^{-1}, and also demonstrates the good empirical performances of its routines on datasets from LibSVM (in Supplementary materials E).  Both linear regression and volume sampling are well studied topics, and the observations made in this paper are quite surprising. The paper clearly outlines a class of matrices that are problematic for volume sampling, and then proves the properties of the revised methods. The proposed methods also exhibit significant empirical gains over other methods in the small sample size regime, which are arguable the more important cases. I believe these contributions are of significant interest to the study of both randomized sampling and randomized numerical linear algebra.


The Nearest Neighbor Information Estimator is Adaptively Near Minimax Rate-Optimal
We analyze the Kozachenko-Leonenko (KL) fixed k-nearest neighbor estimator for the differential entropy. We obtain the first uniform upper bound on its performance for any fixed k over Hölder balls on a torus without assuming any conditions on how close the density could be from zero. Accompanying a recent minimax lower bound over the Hölder ball, we show that the KL estimator for any fixed k is achieving the minimax rates up to logarithmic factors without cognizance of the smoothness parameter s of the Hölder ball for s ∈ (0, 2] and arbitrary dimension d, rendering it the first estimator that provably satisfies this property.

Paper 1614  This paper studies the Kozachenko-Leonenko estimator for the differential entropy of a multivariate smooth density that satisfy a periodic boundary condition; an equivalent way to state the condition is to let the density be defined on the [0,1]^d-torus. The authors show that the K-L estimator achieves a rate of convergence that is optimal up to poly-log factors.   The result is interesting and the paper is well-written. I could not check the entirety of the proof but the parts I checked are correct. I recommend that the paper be accepted.   Some questions and remarks:  * The periodic boundary condition is unnatural, though one can imagine that the problem is much harder without it. Can the authors comment on whether anything can be shown for densities over R^d?  * Can the weighted KL estimator proposed in [1] be used to control the bias and handle cases where s > 2 ?   * It seems that the Besicovitch (typo on line 99) covering lemma implies the Hardy-Littlewood maximal inequality. I only see the variance bound use the former so perhaps the latter result need not be mentioned. On an unrelated note, I am curious whether the authors know what the dependence of the C_d constant on dimensionality d is in the Besicovitch covering lemma.   * Can similar estimator and analysis be used for estimating the Renyi entropy?  [1] T. Berrett, R. Samworth, and M. Yuan, Efficient multivariate entropy estimation via nearest k-neighbor estimators.


Smoothed Analysis of Discrete Tensor Decomposition and Assemblies of Neurons
We analyze linear independence of rank one tensors produced by tensor powers of randomly perturbed vectors. This enables efficient decomposition of sums of high-order tensors. Our analysis builds upon Bhaskara et al. [3] but allows for a wider range of perturbation models, including discrete ones. We give an application to recovering assemblies of neurons. Assemblies are large sets of neurons representing specific memories or concepts. The size of the intersection of two assemblies has been shown in experiments to represent the extent to which these memories co-occur or these concepts are related; the phenomenon is called association of assemblies. This suggests that an animal's memory is a complex web of associations, and poses the problem of recovering this representation from cognitive data. Motivated by this problem, we study the following more general question: Can we reconstruct the Venn diagram of a family of sets, given the sizes of their`-wise intersections? We show that as long as the family of sets is randomly perturbed, it is enough for the number of measurements to be polynomially larger than the number of nonempty regions of the Venn diagram to fully reconstruct the diagram.

# Review for "Smoothed Analysis of Discrete Tensor Decomposition and Assemblies of Neurons"  This paper models and studies the following problem. The brain is known to store "concepts" (the Eiffel tower, Barak Obama) as collections of neurons which fire simultaneously when the concept is "sensed" -- e.g. when one sees a picture of Barak Obama. Furthermore, if there are associations among concepts -- e.g. one has seen a picture of Barak Obama standing in front of the Eiffel tower -- then the "Barak Obama" neurons will tend to fire with the Eiffel tower ones. The high-level question is: is it possible to efficiently learn a "concept map" of someone's brain by measuring strength of associations among small sets of concepts?  The authors model this question as one of learning a set system from (i.e. the Venn diagram of a collection of sets S_1...S_n over some universe) from the sizes of all $\ell$-wise intersections, for some "reasonable" $\ell$. The main result is a polynomial-time algorithm accomplishing this learning problem, so long as  1. set memberships are slightly randomly perturbed (a la smoothed analysis) and 2. the number of measurements $n^\ell$ is larger than the size of the underlying universe.  The algorithm assembles these $n^\ell$ measurements into a certain $\ell$-tensor, then applies known tensor decomposition algorithms to recover the set system. The catch is that these tensor decomposition results are only known to apply to in a smoothed-analysis setting with *Gaussian* perturbations to the underlying tensor components. This is not a natural kind of perturbation in this discrete, set-system setting. The authors show that these tensor decomposition algorithms also work under a much wider variety of perturbation models, essentially any random perturbation for which the each coordinate's perturbation remains slightly anticoncentrated even conditioned on the perturbations of all other coordinates. This technical result on allowable perturabtion models for smoothed analysis is of interest in its own right -- smoothed analyses for high-dimensional geometric problems are few and far between because they pose significant technical challenges, and the authors seem to have invented some interesting new tools to carry out their analysis.  I am not an expert in this kind of neural modeling, so I do not know whether similar models and algorithmic questions have been addressed in prior literature. I am an expert in tensor decomposition algorithms; in my view the new smoothed analysis represents a NIPS-worth contribution in itself, especially since the application is reasonably interesting. Tensor decomposition is a major tool in provable algorithms for machine learning: expanding the class of smoothed analyses under which tensor decomposition methods are known to work removes unnecessary and unnatural technical assumptions on the kinds of models which can be learned by tensor decomposition.  The writing is clear in the first few pages; I did not have the chance to verify the technical details line-by-line but the approach described seems reasonable. I recommend in favor of accepting this paper to NIPS.


BRUNO: A Deep Recurrent Model for Exchangeable Data
We present a novel model architecture which leverages deep learning tools to perform exact Bayesian inference on sets of high dimensional, complex observations. Our model is provably exchangeable, meaning that the joint distribution over observations is invariant under permutation: this property lies at the heart of Bayesian inference. The model does not require variational approximations to train, and new samples can be generated conditional on previous samples, with cost linear in the size of the conditioning set. The advantages of our architecture are demonstrated on learning tasks that require generalisation from short observed sequences while modelling sequence variability, such as conditional image generation, few-shot learning, and anomaly detection.

[After author rebuttal]  My thanks to the authors for their careful rebuttal. I found it convincing & I've modified my review to a weak accept. Original review below.  [Original review]  This paper proposes a model for exchangeable data based on sampling latent variables from an exchangeable student T process, followed by passing the variables through an easily-invertible (NVP) transformation. The paper proposes an efficiently-computable decomposition of the proposed distribution, and provides experimental evidence of its utility.  This paper is reasonably well-written and clearly explained. The intro and relevant work sections are particularly well done. Some comments for improvements in the rest of the paper: -- section 3.3: the generative model is described backwards, i.e. x first (with x exchangeable), then a mapping from z to x (z exch), then the distribution on z. I would think it would be easier to understand if the distribution on z is described, then how z maps to observed data x, then show x is exchangeable -- from l.173-183, it's clear that the paper derives the quantities needed to express p(z_{n+1} | z_{1:n}), but it's not clear why that distribution is needed or useful -- section 3.4 makes it clearer that the efficient expression of p( *_n+1 | *_{1:n} ) is used for training, where the goal is to maximize the log marginal p(x_1, ... x_n) (decomposed as the sequence of conditionals).  -- I think the right way to fix this confusion is to move section 3.4 earlier in 3.3, then expand on it. That is, state around l.168 that given the specification of the model, the major task is to train it. Then explicitly write the training problem of maximizing the log marginal, then decompose it, then show that the student T process yields a nice efficient formula for the decomposition, then argue this can be optimized (via e.g. autograd). Finally, mention that the efficient conditional formula can be used for prediction once training is done. It would also be very helpful to write down the full model including the NVP transformation and how it affects training. -- A similar clarity issue occurs in the experiments. It's generally unclear how data is being used for training & testing/prediction. Is the idea that you take a bunch of sequences of length N and optimize the sum of log marginals for each (  so maximize sum_{j=1}^J sum_{n=1}^{N-1} log p(x_{j,n+1} | x_{j,1:n}) where each (x_{j,1}, ... , x{j, N}) is an observed sequence )?  -- I'm not sure what I'm supposed to get from figure 2. Aside from being too small (given the intricate characters), it is missing a lot of explanation. What is an "input sequence"? Were these samples generated on a model trained on the whole dataset or a single sequence? Is this showing samples of p(x_{N+1}, ... x_{2N} | x_{1:N}) where x_{1:N} are shown in the top row? Is this all simulated from the model? I really am not sure what I'm seeing here. -- I'm also not convinced that I see "lower variability in appearance" for images in the right col -- the experiment in 4.2 is a bit clearer, but it's still missing details on how the model was trained. Is this the same model from 4.1? -- the extra "discriminative training" is not clear and not well-described.  - a technical question: Is there a known de Finetti representation for the exchangeable student T (similar to that for the gaussian at l.37)? If one were to explicitly incorporate this top hierarchical layer into the model, would that make inference more efficient (given that the z's would then be conditionally independent)?  Given the missing/unclear details in section 3 and the experiments, I've given this a weak reject. It's very borderline though; I'd certainly be open to switching to an accept given a convincing rebuttal.  Detailed comments: - l.2 - the paper asserts it performs exact bayesian inference. I would hesitate to make this claim, given that the paper marginalizes out the latent parameter - l.12-16: exchangeable sequences are not a model for unordered sets, but rather for ordered sequences where the probability of generating a particular sequence is invariant to reordering. Sequences of unordered sets (and derivative combinatorial structures) need not be exchangeable, although sometimes are. cf Pitman, "exchangeable and partially exchangeable random partitions" and Broderick, Pitman, & Jordan "feature allocations, probability functions, and paintboxes"  - Figure 1 should appear later in the paper, perhaps at the beginning of section 3.3.  - mathcal script font shouldn't be used for TP / GP. Just use regular text for these initialisms - l.105: "nontrivial differences in the task" - better to just be clear here about exactly why they're different and what benefit TP gives over GP - l.130: In particular it's the marginalization over the inverse Wishart covariance matrix - l.134: NVP is undefined - l.150-153: is the uniform noise discussion really necessary? Why not just ignore the discreteness and treat the quantized data as real-valued in practice? - l.163+: generally I'd just use two subscripts rather than superscript - too easily confused with exponentiation (especially in statements like K_{ii}^d = v^d)


Completing State Representations using Spectral Learning
A central problem in dynamical system modeling is state discovery-that is, finding a compact summary of the past that captures the information needed to predict the future. Predictive State Representations (PSRs) enable clever spectral methods for state discovery; however, while consistent in the limit of infinite data, these methods often suffer from poor performance in the low data regime. In this paper we develop a novel algorithm for incorporating domain knowledge, in the form of an imperfect state representation, as side information to speed spectral learning for PSRs. We prove theoretical results characterizing the relevance of a user-provided state representation, and design spectral algorithms that can take advantage of a relevant representation. Our algorithm utilizes principal angles to extract the relevant components of the representation, and is robust to misspecification. Empirical evaluation on synthetic HMMs, an aircraft identification domain, and a gene splice dataset shows that, even with weak domain knowledge, the algorithm can significantly outperform standard PSR learning.

SUMMARY:  This paper proposes a method to incorporate prior knowledge into the spectral learning algorithm for predictive state representations (PSR). The prior knowledge consists of an imperfect/incomplete state representation which is 'refined' and 'completed' by the learning algorithm. This contribution addresses one of the main caveats of spectral methods: while these methods are fast and consistent, they tend to perform worse than local methods (e.g. EM) in the low data regime. By leveraging domain specific knowledge, the proposed algorithm overcomes this issue.  The proposed extension, PSR-f, is relatively straightforward: the belief vector at each time step is the concatenation of the user-specified state representation f with a learned state representation b; the parameters of b are learned in the same fashion as for the classical method by solving linear regression problems constrained to the row space of the concatenation of some Hankel/system matrices (e.g. now mapping [P(T | h) ; P(h)f(h)] to P(oT | h) for each B_o). The authors then give sufficient conditions for the algorithm to be consistent and formally analyze to which extent providing prior information to the algorithm can be beneficial or detrimental; indeed, the number of states of a minimal PSR-f can be greater than the one of the corresponding minimal PSR when f is 'irrelevant'! To circumvent this issue, the authors proposed a method to extract the relevant information from f before performing the spectral learning algorithm by analyzing the principal angles between the classical Hankel matrix and some feature matrix induced by the user-specified state representation f.  The proposed method is evaluated on both synthetic and real data experiments where PSR-f is compared with the basic PSR spectral learning algorithm (i.e. not using the prior knowledge of f) and with only using the user-specified state representation f.   OPINION:  I liked the paper, it is well written and relatively easy to follow for someone familiar with the spectral learning literature. As mentioned above, I think that the problem addressed in the paper is relevant and the solution proposed by the author is natural and sound (I did not check the maths thoroughly but the results stated in the Theorems seems accurate and expected), and that this contribution is a nice addition to the spectral learning literature. The paper could maybe benefit from a more high-level / abstract exposition of the technical material, e.g. reasoning also in terms of subspaces rather than only with the Hankel sub-blocks in Defs 2 to 5 (see my comment below), but this may be a matter of personal taste.     REMARKS/QUESTIONS:  - I find it strange to introduce the \beta_o as parameters, is it standard notation? I am more familiar with having a normalizer  beta_\infty as a parameter of the model instead and defining the update equation with \beta_\infty B_o b(h) at the denominator (see e.g. [4]). How are the two related?  - Is the requirement that the core tests and histories are minimal necessary for the consistency results?   - In definitions 2 and 3, I believe that H (and even T) can be replaced directly with O^* instead of having a universal quantifier and a sup respectively. Then, in my opinion, defining rank(f;M) as the dimension of the intersection of the row spaces of P_{O^*,O^*} and P_{f,O^*} is simpler and more intuitive (this is actually what is indirectly done in Prop. 7 in the supplementary). Indeed, this intersection is the smallest subspace of P_{f,O^*} that is contained in P_{O^*,O^*}, i.e. that is linearly relevant to M.  Then the matrix U_f in definition 4 can be understood as some projection onto this intersection (i.e. the rows of U_f^T U_f P_{f,H} belongs both to the row spaces of P_{O^*,H} and P_{f,H}).  Finally, Theorem 2 simply follows from the formulas for the dimension of the complement of P_{O^*,H} \cap P_{f,H} in P_{O^*,H} (for |T|), and for the dimension of the sum P_{O^*,H} + P_{f,H} (for |H|). Now, similarly to classical PSR, the minimal sizes for T and H are equal to rank of the concatenation of P_{O^*,H} and P_{f,H}, which can be as low as rank(M) but as high as rank(M) + n (when the intersection of  P_{O^*,H} and P_{f,H} is {0}).  - I find the phrasing "solving by matrix inverse" a bit odd.    MINOR COMMENTS/TYPOS:  - line 52: since there are no restriction on Z line 48, setting Z = R^n and assuming that P(o|h) is linear in the belief is actually a restriction rather than a generalization.  - line 76: R -> R^m - line 113-114: use (resp. H) ... (reps. T). - line 136: spurious ) after H - line 158: paramter -> parameter


Measures of distortion for machine learning
Given data from a general metric space, one of the standard machine learning pipelines is to first embed the data into a Euclidean space and subsequently apply machine learning algorithms to analyze the data. The quality of such an embedding is typically described in terms of a distortion measure. In this paper, we show that many of the existing distortion measures behave in an undesired way, when considered from a machine learning point of view. We investigate desirable properties of distortion measures and formally prove that most of the existing measures fail to satisfy these properties. These theoretical findings are supported by simulations, which for example demonstrate that existing distortion measures are not robust to noise or outliers and cannot serve as good indicators for classification accuracy. As an alternative, we suggest a new measure of distortion, called σ-distortion. We can show both in theory and in experiments that it satisfies all desirable properties and is a better candidate to evaluate distortion in the context of machine learning.

=== SUMMARY === When points in one metric space are embedded into another (eg. from high to lower dimension, or from a "difficult" to "easy" metric), the quality of the embedding is typically measured by some notion of distance distortion. This paper is a systematic study of distortion measures. It formally defines several desired properties of a distortion measure, and compares existing distortion measures from the lens of those formal definitions and by simulations on synthetic data. Based on these, the pros and cons of each measure are discussed, and a new notion of distortion is suggested.  === COMMENTS === The systematic study of desirable properties of distortion is solid and in my view constitutes the main strength of this submission. The discussion leading to the formal definitions is clear and well-motivated, and the definitions are well-thought and display theoretical expertise. The findings, both in theory and in simulation, are interesting and for me they shed light on how to interpret embedding results. The new sigma-distortion is simple and straightforward, but it is interesting to see both rigorously and experimentally how it compares favorably to existing distortion measures (which are also simple and straightforward, but apparently behave less desirably). Since the rest of my review will be mostly critical, I will say in advance that the strengths mentioned so far are sufficient for me to view this paper as a solid contribution that can be accepted.  The main weakness of the paper is a somewhat tenuous connection to machine learning in terms of motivation, application and literature.  Motivation: While the paper claims to be about ML, the relevance to ML is described only superficially and in broad strokes. It is true that the standard ML pipeline embeds the data into Euclidean space, but mostly the domain space is not metric, but a space of "real-world objects" (images, text documents etc) with no a-priori well-defined or quantifiable measure of distance, and therefore the embedding cannot be evaluated in terms of distortion (this "embedding" is actually feature generation). Distortion measures are relevant specifically for dimensionality reduction, either to achieve computational tractability or when the data is assumed to have low intrinsic dimension despite its high ambient dimension, which is quite a common assumption and well-observed phenomenon. Another possibly relevant case is when the data is endowed with a distance function, but it is "difficult" and one wants to embed it into a nicer space like Euclidean which is easier to work with. I am not sure I am giving a good overview of the ML motivation; my point is that being more specific about the relevance of the subject (preferably with appropriate citations) would help the paper.  Literature: The introduction and the list of allegedly "popular" distortion measures cite prior theoretical work, but not any ML work at all. (Are there any ML citations before the experimental section?) It seems that the authors are well-versed in the mathematical literature, but could have made more effort to connect their work to the ML literature, which is important for NIPS.  Application: Distortion measures are used in theoretical analysis, but It is not clear how they arise in application at all. It is too costly to compute the distortion of a given embedding for assessing its quality, and doing this to choose between candidate embeddings seems unlikely in practice. If anything, distortion measures are used in designing objective functions to find a good embedding between two explicit spaces (eg. dimension reduction), and this raises computational issues (eg. how to efficiently optimize over them) which are not mentioned in this paper. The embedding results in theorems 3,4 appear to be of the theoretical kind that offer qualitative insight but have no bearing in practice.  One example work in this vein is "Spectral Hashing" (Weiss, Torralba, Fergus, NIPS'08). Variants of the "spectral" distortion they use as objective (their eq.(3)) are common in many other ML works as well. I am not sure if it squarely fits into one of the distortion measures discussed here (it seems to be a form of l_2-distortion, but I did not check in detail), but it looks related at least conceptually.  The experiments utilize only synthetic data, but I find it a reasonable choice in the context of the paper.  In terms of clarity, The paper is well written and organized and was a pleasant read. One point I would consider making clearer is doubling dimension, which is mentioned several times but is not properly explained. The formal definition might be unsuitable for the main text, but a short explanation about intrinsic dimension could help, and some relevant citations could help more (eg. "Efficient Classification for Metric Data" (Gottlieb, Kontorovich, Krauthgamer, COLT'10)).  The supplementary PDF seems to be corrupted -- I could not open it despite several attempts on different machines. From the main text I gather that it contains the proofs of the theorems and perhaps additional simulation results. My understanding is that the proofs are either relatively straightforward given the definitions (theorems 1 and 2) or mostly follow from prior work (theorems 3 and 4). The simulations included in the main file are sufficiently complete to make the point they claim to make. Overall the main text stands well on its own. I am therefore willing to disregard the unavailability of the supplementary file for the sake of this review (and in particular to tentatively believe the claimed theorems).  Question: Are there direct connections between the notions of distortion? Eg. does epsilon worst-case distortion imply (say) epsilon l_p-distortion of sigma-distortion? This could help make sense of the embeddability results.  === CONCLUSION === Pros: I find the systematic study of distortion measures to be interesting and valuable. The discussion and formalization of desirable distortion properties is thorough and well-thought. The formal and experimental comparison is well-established and yields interesting insights. Overall the paper makes a solid and worthwhile contribution.  Cons: The motivation and relevance of this work to machine learning remains somewhat tenuous. The introduction could use a better discussion of the motivation, and relevant prior work in the machine learning literature could be better surveyed. I do think this paper is in scope for NIPS, and that the connection could be established convincingly with some effort on part of the authors. I hope they somewhat improve this point in the final version, as it will help make their work accessible and appealing to the audience in the venue they chose to target.  *** Update after author response: I am somewhat surprised by the firm practicality claims in the rebuttal, and still not quite convinced. The authors write that even "simply" reading the distance matrix takes quadratic time; true, but there is rarely justification for fully reading (nor generating to begin with) the whole distance matrix, and quadratic time is largely considered prohibitive or at least undesirable for even moderate size data. Perhaps it makes sense for the setting the authors have in mind, and this ties back to the issue of better explaining the context of their work. The conclusion of my review remains the same.


A Block Coordinate Ascent Algorithm for Mean-Variance Optimization
Risk management in dynamic decision problems is a primary concern in many fields, including financial investment, autonomous driving, and healthcare. The mean-variance function is one of the most widely used objective functions in risk management due to its simplicity and interpretability. Existing algorithms for mean-variance optimization are based on multi-time-scale stochastic approximation, whose learning rate schedules are often hard to tune, and have only asymptotic convergence proof. In this paper, we develop a model-free policy search framework for mean-variance optimization with finite-sample error bound analysis (to local optima). Our starting point is a reformulation of the original mean-variance function with its Legendre-Fenchel dual, from which we propose a stochastic block coordinate ascent policy search algorithm. Both the asymptotic convergence guarantee of the last iteration's solution and the convergence rate of the randomly picked solution are provided, and their applicability is demonstrated on several benchmark domains.

The paper "A Block Coordinate Ascent Algorithm for Mean-Variance Optimization" proposes a new algorithm for reinforcement learning with a mean-variance objective that has a hard constraint on the reward variance and analyses the convergence of the algorithm in both infinite and finite horizon cases. The derivations for the new algorithm use a transformation based on a Fenchel dual (to transform a quadratic term into a linear term plus terms with an added additional variable) enabling gradient descent to be applied on the transformed Lagrangian dual of the original objective. The contribution w.r.t. the proofs and the new algorithm appear solid.  The paper is in general well written. There are some typos that should be fixed. The main problem in the paper is the reporting of the experimental results. The representation used for the policy is not discussed, the variance limit is not defined, and it is unclear whether the reported results are statistically significant.  The main claims about related work are as follows: "1) Most of the analyses of ODE-based methods are asymptotic, with no sample complexity analysis." Most are asymptotic? The paper needs to discuss those cases where the analysis is not asymptotic. This is discussed later in the paper? Should be already mentioned here. "2) It is well-known that multi-time-scale approaches are sensitive to the choice of the stepsize schedules, which is a non-trivial burden in real-world problems." "3) The ODE approach does not allow extra penalty functions. Adding penalty functions can often strengthen the robustness of the algorithm, encourages sparsity and incorporates prior knowledge into the problem [Hastie et al., 2001]."  Claims 1) - 3) seem correct and the proposed algorithm could offer some improvement over these previous approaches.  Algorithm 1: Each episode starts from the initial state s_1 until ending in the recurrent state s^*. It could help the reader to give an explanation why this is done so? Is this absolutely necessary for the analysis of the algorithm?  EXPERIMENTS: In the experiments, it is unclear what kind of representation the policy follows. It cannot be tabular since some of the problems have continuous states. It is also unclear whether the form of the policy influences the performance of the algorithms.  \xi, the limit on the expected variance of the rewards (Equation 1), should be defined for each problem.  The statistical significance of the results is now described only in textual form. For example, "The results indicate that MVP yields a higher mean return with less variance compared to the competing algorithms."  Firstly, according to my understanding the objective is not to minimize variance but to limit it, at least according to Equation 1. As such the variance of the comparison methods does not matter if it remains under the limit (which is not defined). Of course the variance is of separate interest for understanding how far each algorithm is from the limit. I recommend reporting whether the variance limit is exceeded and by how much. Furthermore, I recommend performing statistical significance testing on both the mean and variance.  LANGUAGE: In some places there is some unclarity that should be fixed. For example, in "yet almost all the reported analysis of these algorithms are asymptotic [Xu and Yin, 2015]." almost all? Which reported analysis is not asymptotic?  "In risk-sensitive mean-variance optimization MDPs, the objective is often to maximize J(θ) with a variance constraint, i.e.," *often* ? Earlier in the paper there is a discussion on related work but it was not made clear what exactly the objective in those papers is?  Smaller typos: Line 85: "the third time-scale is to optimizes over" -> the third time-scale optimizes over Line 48: "2 Backgrounds" -> "2 Background" ? Line 116: "and derives novel algorithms" -> and derive novel algorithms Line 188: "for which there is no established results" -> for which there are no established results Line 203: "Without the loss of generality" should be usually "Without loss of generality" Line 205: "as as" -> as Line 206: "approximate error" -> approximation error Line 219: "Now it is ready to provide" -> Now we provide


ATOMO: Communication-efficient Learning via Atomic Sparsification
Distributed model training suffers from communication overheads due to frequent gradient updates transmitted between compute nodes. To mitigate these overheads, several studies propose the use of sparsified stochastic gradients. We argue that these are facets of a general sparsification method that can operate on any possible atomic decomposition. Notable examples include element-wise, singular value, and Fourier decompositions. We present ATOMO, a general framework for atomic sparsification of stochastic gradients. Given a gradient, an atomic decomposition, and a sparsity budget, ATOMO gives a random unbiased sparsification of the atoms minimizing variance. We show that recent methods such as QSGD and TernGrad are special cases of ATOMO and that sparsifiying the singular value decomposition of neural networks gradients, rather than their coordinates, can lead to significantly faster distributed training.

After rebutal; I do not wish to change my evaluation. Regarding convergence, I think that this should be clarified in the paper, to at least ensure that this is not producting divergent sequences under resaonable assumptions. As for the variance, the author control the variance of a certain variable \hat{g} given g but they should control the variance of \hat{g} without conditioning to invoke general convergence results. This is very minor but should be mentioned.      The authors consider the problem of empirical risk minimization using a distributed stochastic gradient descent algorithm. In this setting, communication cost constitutes a significant bottleneck. The authors generalize recently proposed scheme which aims at obtaining random sparse unbiased estimates of a vector while ensuring minimal variance. The authors describe an active set algorithm which allows to perform this operation given decomposition in an orthonormal basis. Extensive experiments based on low rank approximation of the gradient matrix suggest that the proposed approach allows to speedup convergence by reducing the communication cost.   Main comments I have very limited knowledge about distributed optimization, so I might miss some important points or shortcomings in the paper.  Overall the paper is well written and seems to reference correctly the corresponding litterature. The proposed algorithm is quite natural. I did not have a careful look at all proof details but I could not see anything which looked abnormal. I am not really able to criticize numerical experiments beyond the fact that they are convincing to me.  The overall mathematical exposition is quite elementary, some elements could be cut out, for example all the Lemmas related to algorithm 1 may not be necessary in the main text, this is actually a simple active set method. Another example is the notion of equivalence of norms and Theorem 10, this can be found in a first year analysis course and does not bring much to the paper. Lemma 8 is also well known.   The question of the convergence of this process is not touched. How does the sparsification affect convergence?  Why not comparing to ternGrad?    Minor comments  I am not quite sure about the relevance of the concept of "atomic decomposition". From definition 1, this is just decomposition in an orthonormal basis, which is a less fancy name but more standard as a mathematical concept.  In Theorem 5, what does $j = argmax$ mean in the second statement? Similarly what is "i" in the first statement. Logical quantifiers should be used here.  Figure 1, I would rather say top 2 values standing out  The constraint "can be represented with k bits" is not really meaningful. Any character string can be compressed to a single bit using the appropriate dictionary. May be the authors have in mind a specific representation or model.  "In what follows, you may think of g as the stochastic gradient" is not at the usual level of formality in scientific papers.  Line 203: we s and suppose


LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning
This paper presents a new class of gradient methods for distributed machine learning that adaptively skip the gradient calculations to learn with reduced communication and computation. Simple rules are designed to detect slowly-varying gradients and, therefore, trigger the reuse of outdated gradients. The resultant gradient-based algorithms are termed Lazily Aggregated Gradient -justifying our acronym LAG used henceforth. Theoretically, the merits of this contribution are: i) the convergence rate is the same as batch gradient descent in stronglyconvex, convex, and nonconvex cases; and, ii) if the distributed datasets are heterogeneous (quantified by certain measurable constants), the communication rounds needed to achieve a targeted accuracy are reduced thanks to the adaptive reuse of lagged gradients. Numerical experiments on both synthetic and real data corroborate a significant communication reduction compared to alternatives.

This paper explores the question of minimizing the communication among workers while solving an optimization problem in a distributed fashion. In particular, the authors argue that most of the existing work in this direction focused on minimizing the amount of data during each message exchange between the works. In contrast, the authors focus on reducing the number of such exchanges (or communication rounds) during the optimization procedures. The authors argue that reducing the number of rounds is more beneficial as it does not degrade the rate of convergence for a wide class of objective functions.  The authors propose a simple approach to reduce the communication: a worker sends the current values of the gradient (based on its data) only if this gradient is significantly far from the previous gradient supplied by this worker, leading to the name lazily aggregated gradient (LAG). The authors propose two variants of this approach depending on whether the parameter server or the individual worker decides when to send the current estimate of the gradient. Under natural assumptions of the objective functions, such as smoothness, convexity, or strong-convexity, the authors show that their approach results into the convergence rate similar to that of the conventional gradient descent method where each worker sends the current estimate of the gradient (based on its data) during each iteration. Using a detailed experimental evaluation, the authors show that the LAG approach significantly outperforms some of the previous approaches to reduce the communication in terms of both the convergence rate and the total amount of communication among the workers.   The paper proposes a simple yet effective method to reduce the overall communication load for a distributed optimization problem. The proposed method has favorable performance guarantees and has been shown work well with both synthetic and real datasets.   The authors argue that the proposed solution can be combined with other approaches to reduce communication, such as quantized gradients and second-order methods. Can authors comment on how the theoretical guarantees might be affected by bringing in these additional ideas?


Reinforcement Learning for Solving the Vehicle Routing Problem
We present an end-to-end framework for solving the Vehicle Routing Problem (VRP) using reinforcement learning. In this approach, we train a single policy model that finds near-optimal solutions for a broad range of problem instances of similar size, only by observing the reward signals and following feasibility rules. We consider a parameterized stochastic policy, and by applying a policy gradient algorithm to optimize its parameters, the trained model produces the solution as a sequence of consecutive actions in real time, without the need to re-train for every new problem instance. On capacitated VRP, our approach outperforms classical heuristics and Google's OR-Tools on medium-sized instances in solution quality with comparable computation time (after training). We demonstrate how our approach can handle problems with split delivery and explore the effect of such deliveries on the solution quality. Our proposed framework can be applied to other variants of the VRP such as the stochastic VRP, and has the potential to be applied more generally to combinatorial optimization problems.

Updated review: ---------------------- I am happy with the authors response to my two main concerns. I am raising my score to an accept conditional on the additional results being added to the paper.  Original review: --------------------- The reviewed paper presents an application of model-free reinforcement learning to the problem of optimal vehicle routing. The considered application is interesting and potentially of high-impact. It is computationally difficult to solve and finding optimal solutions for large instances is - "in the wild" - currently done with the use of heuristics. The presented appproach marks a significant improvement over said heuristics while being "cheap" to compute.   The paper is well written and easy to follow, the related work background is reasonable, yet it fails to point out connections to graph-networks. The description of problem setting is thorough and easily understandable, making this paper a good reference for future attempts to address similar problems with ML techniques.  The contribution of the paper is purely empirical - which is not necessarily a bad thing. It builds on existing theory and proposes problem specific adjustments. The empirical evaluation is thorough and meaningful, giving useful comparisons to externally implemented SOTA algorithms as well as to simple heuristics.  Arguably such a paper might be better suited for presentation in an operations research conference or journal and not perfectly suited for NIPS. I however refrain from such judgement as I think the paper is interesting enough (modulo my main concerns below) and would suggest that the area chair has a second look.  Overall the paper could well be accepted to NIPS if the two main concerns that I outline below would prove to be unsubstantiated / could be addressed. For now I mark it with a weak reject but I would increase my score to a weak accept/accept if the concerns can be addressed (which would merely require running additional experiments with Pointer networks and adding a short paragraph in the related work).  Main concerns with the paper: 1) Section 3.1 makes it sound as if pointer networks are simply not applicable to the problem. Which is not the case as far as I understand. Sure it will be more expensive to re-evaluate the pointer network once an input has changed but it is technically possible. I fail to see why this cannot be implemented (and in fact just changing the inputs every time step e.g. in tensorflow should automatically generate the required computational graph). In fact you mention in l. 189 that the problem with applying  a simple Pointer Network is mainly a computational one. If this is the case then you should provide a comparison to pointer networks at least on small scale instances and show when and how they break!  2) Your proposed solution in 3.2, to me, reads simply as applying a special case of a graph network (see e.g. [1] and maybe [2-3] for introductions) how is this different than a graph network with a pointer network output ? If it is not then you should discuss/relate to the graph-network line of work in the related work/background section.   Experiments:  - What is the variance of the results wrt. training seeds ? I.e. the plot in Figure 3 seems to show variance accross problem instances for one trained policy, but how does the performance vary wrt. the training itself ? - One appealing property of the method should be that it can handle variable problem sizes yet no attempt is made to quantify this property. Why do you not  try to train on VRP10 to VRP100 and see how well a joint model does ? Would such variability in training sequence size not anyway be required for any practical application ? If so it would also be important to know how the policy compares in these instances to e.g. OR-Tools. Unfortunately does not generalize over problem instances, e.g. more nodes etc.   Specific comments and minor typos: l. 29.: "milestone on the path toward approaching" -> I guess you mean milestone on the path towards solving ? l. 34-36: "Obviously, this approach is not practical in terms of either solution quality or runtime since there should be many trajectories sampled from one MDP to be able to produce a near-optimal solution." -> It is not clear to me what you mean here, I guess you mean that solving an MDP "on-the-fly" every time demands change is too expensive ? What do you mean by many trajectories are needed ? It is not clear that you could not use a model for this to me and what the exact difference is you are trying to point out here. Revise that sentence. l. 52: "which is actually simpler than the Pointer Network approach" -> Remove actually. This sounds as if there was a counter-point to said claim before. Expressing your surprise here is not necessary. l. 54: "Our model" -> it is confusing to call this a model here and before as you are *not* trying to learn a model of the MDP but just use a different parameterization for the policy. Maybe re-write to "policy model" or "actor-critic model" or something like that ?  l. 59: appealing to whom ? I guess you mean it is appealing to practitioners ? l. 64: "inputs change in any way" -> clearly not in any arbitrary way :) try to be a bit more precise, e.g. do you want to say it is robust to noise or to changing rewards of the VRP or ... l. 72: "is ones" -> "is one" l. 78: "and *at* no extra cost" l. 80: reads colloquial better: "[...] we briefly review required notation and relations to existing work" ?  l. 84: "which is almost the same": either substantiate this claim or simply write: "which shared by many of these models" l. 91: "wisely" is inappropriate here -> chang eto "more directly" or "more explicitly" l. 98: "for policy representation" -> "for the policy parameterization" Section 3: as written above the use of the term Model is a bit loose in the paper. Sometimes it referes to the policy, somethimes to the problem formulation itself (the MDP) and sometimes to the combination of both, try to be more specific. l. 226: remove braces this is a full sentence.  [1] Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., and Monfardini, G. Computational capabilities of graph neural networks [2] Li, Y., Tarlow, D., Brockschmidt, M., and Zemel, R. Gated graph sequence neural networks.  [3] Battaglia, P., Pascanu, R., Lai, M., Rezende, D. J., et al. Interaction networks for learning about objects, relations and physics


Online convex optimization for cumulative constraints
We propose the algorithms for online convex optimization which lead to cumulative squared constraint violations of the form
β ∈ (0, 1) . Previous literature has focused on long-term constraints of the form
g(x t ). There, strictly feasible solutions can cancel out the effects of violated constraints. In contrast, the new form heavily penalizes large constraint violations and cancellation effects cannot occur. Furthermore, useful bounds on the single step constraint violation [g(x t )] + are derived. For convex objectives, our regret bounds generalize existing bounds, and for strongly convex objectives we give improved regret bounds. In numerical experiments, we show that our algorithm closely follows the constraint boundary leading to low cumulative violation.

The paper studies the usual online convex optimization framework with the additional twist that the algorithm can predict a point outside of the decision set. The decision set is specified by a set of convex constraints g_i(x) <= 0, i=1,2,...,m.  The goal of the algorithm is to achieve small regret with respect to the best point the decision set and at the same time achieve small sum of constraint violations. A constraint violation is penalized with the square, (g_i(x))^2.  The motivation for this somewhat artificial looking problem is to avoid projections to the decision set in every round, which would be otherwise required.  The paper attacks the problem via Lagrangian relaxation. The resulting algorithm is primal-dual algorithm that operates both on primal variables as well the Lagrangian variables. The resulting algorithm is an algorithm similar to subgradient descent that at the same time adjusts the Lagrangian variables.  The paper proves that the algorithm has O(sqrt(T)) regret and O(sqrt(T)) sum of constraint violations in the case when both the cost functions and functions specifying the constraints are (convex) and Lipchitz.  In the case when the cost functions are strongly convex and constraints are Lipchitz, the proves O(log(T)) regret bound and a bound O(sqrt(log(T) * T)) on sum of constraint violations.  ---  Suggestions for improvement:  1) In the introduction (Lines 29-32) g(x_t) is used without first explaining that S is expressed as S = {x : g_i(x) <= 0, i=1,2,...,m}. This is confusing and the reader is left wondering where g comes from.  2) Use a different letters of the alphabet for the unit ball \mathbb{B} and the enlarged set \mathcal{B}; different fonts are not good enough distinction. I overlooked the definition of \mathcal{B} line 70 and was left wondering if it is a typo.  3) Line 102: max --> \max  4) Line 162: argmin --> \DeclareMathOperator{\argmin}{argmin} \argmin  5) Section 5.2. You have equality constraints in equation (11). Explain how did you transform them into inequality constraints.


Recurrent Transformer Networks for Semantic Correspondence
We present recurrent transformer networks (RTNs) for obtaining dense correspondences between semantically similar images. Our networks accomplish this through an iterative process of estimating spatial transformations between the input images and using these transformations to generate aligned convolutional activations. By directly estimating the transformations between an image pair, rather than employing spatial transformer networks to independently normalize each individual image, we show that greater accuracy can be achieved. This process is conducted in a recursive manner to refine both the transformation estimates and the feature representations. In addition, a technique is presented for weakly-supervised training of RTNs that is based on a proposed classification loss. With RTNs, state-of-the-art performance is attained on several benchmarks for semantic correspondence.

The paper proposes an recurrent iterative refinement approach for weakly supervised semantic correspondence learning between similar objects. The idea is to predict a flow field transformation for each pixel, such that after transforming the image the source and target images are aligned. This paper builds on the weak-supervision similar to [18] and [29] but predicts the flow, and adds an iterative component where in each step the network outputs the residual to that brings the source and the transformed target images closer.  The objective is similar to the concurrent work of [29] (but with negatives) where the idea that transformed pixel j should have the highest feature correlation with the feature at the pixel it got transformed to than any of its immediate neighbors. The paper also proposes a dilated search window for this neighbor radius, where the search window shrinks as the iteration continues such that more finer alignment may be achieved.  The paper evaluates against standard benchmarks, improves upon recent state of the art methods, and obtains visually pleasant results.  Pros of this paper is that: - Results look good - Doing matching in an iterative style makes sense - Progressively shrinking search window makes sense - In a sense it's a simple approach that brings together a lot of the recent   advancements in this field. - Flow rather than affine or TPS allows for finer detail matching.  Weaknesses are: In terms of originality & novelty, while the idea to do iteration has yet to be tried in image matching, this idea has been demonstarted in various other applications:   - Joao Carreira, Pulkit Agrawal, Katerina Fragkiadaki, Jitendra Malik, "Human   Pose Estimation with Iterative Error Feedback", CVPR 2016   - Markus Oberweger, Paul Wohlhart, Vincent Lepetit, "Training a feedback loop     for hand pose estimation" ICCV 2015   - Piotr Dollar, Peter Wlinder, Pietro Perona, "Cascaded pose regression", CVPR     2010  The paper should be more generous in citing these papers.  In an essence this paper could be seen as a combination of [18] and [29] but with flow and iteration + other tricks. Therefore it's not as original, however, [29] is concurrent and given the quality of results I'm inclined towards acceptance. At the same time, this kind of paper might be more suitable for vision conferences like CVPR/ICCV/ECCV.  Authors might want to cite these other related works: - Thewlis et al. Unsupervised learning of object frames by dense equivariant   image labelling. NIPS 2017 Does not learn matching transformation but learns a matchable features, also from synthetic deformations - Kanazawa et al. "WarpNet: Weakly Supervised Matching for Single-view   Reconstruction", one of the first papers to solve for TPS to match two images, trained in a self-supervised manner through synthetic transformation (line 123) and also they demonstrate the matching results for single-view 3D application, relevant to the conclusion of this paper.  Post rebuttal:  I stand by my original rating.


Dendritic cortical microcircuits approximate the backpropagation algorithm
Deep learning has seen remarkable developments over the last years, many of them inspired by neuroscience. However, the main learning mechanism behind these advances -error backpropagation -appears to be at odds with neurobiology. Here, we introduce a multilayer neuronal network model with simplified dendritic compartments in which error-driven synaptic plasticity adapts the network towards a global desired output. In contrast to previous work our model does not require separate phases and synaptic learning is driven by local dendritic prediction errors continuously in time. Such errors originate at apical dendrites and occur due to a mismatch between predictive input from lateral interneurons and activity from actual top-down feedback. Through the use of simple dendritic compartments and different cell-types our model can represent both error and normal activity within a pyramidal neuron. We demonstrate the learning capabilities of the model in regression and classification tasks, and show analytically that it approximates the error backpropagation algorithm. Moreover, our framework is consistent with recent observations of learning between brain areas and the architecture of cortical microcircuits. Overall, we introduce a novel view of learning on dendritic cortical circuits and on how the brain may solve the long-standing synaptic credit assignment problem.

Update after rebuttal:  This is a strong paper. The rebuttal has addressed several of my concerns. ____________________  Summary:  This paper proposes a new biological implementation of an approximate backpropagation algorithm that relies on multi compartment neuron models. Using two compartments allows errors and activities to be represented within the same neuron. The overall procedure is similar to contrastive Hebbian learning and relies on weak top down feedback from an initial ‘self-predicting’ settled state, but unlike contrastive Hebbian learning does not require separate phases. Experimental results show that the method can attain reasonable results on MNIST.  Major comments:  This paper presents an interesting approach to approximately implementing backpropagation that relies on a mixture of dendritic compartments and specific circuitry motifs. This is a fundamentally important topic and the results would likely be of interest to many, even if the specific hypothesis turns out to be incorrect.  The paper is admirably clear and easy to follow.  The method would appear to suffer from a similar difficulty as contrastive Hebbian learning: it only faithfully reflects backpropagation when top down feedback is weak, but this means that weight updates become very small in a deep network. Correcting this by scaling up the weight updates by the inverse of the top down feedback strength is problematic because this would require high precision and would drastically amplify small noise in the neural activity. Most of the experiments presented in the paper focus on shallow, single hidden layer networks. The MNIST experiment with a deeper two hidden layer network might provide an opportunity to test the robustness of the system by showing training curves with frozen first layer weights. More broadly, recent results have shown that schemes like feedback alignment often work well on simple datasets in relatively shallow networks, but struggle to match the performance of backprop in deeper networks on more complex datasets. The paper could be greatly strengthened by testing whether the proposed method really solves the credit assignment problem in substantially deep networks.  In general, the test error rates reported here are typically in the range of what is achievable by a SVM which does not learn its internal representation (e.g., SVM with Gaussian kernel gets ~1.4%). Because of this it is important to show that the learning performance is critically dependent on learning the internal representation. It would be useful to report learning curves with random but frozen first layers (with optimized variance), in addition to the shallow network performance.


Critical initialisation for deep signal propagation in noisy rectifier neural networks
Stochastic regularisation is an important weapon in the arsenal of a deep learning practitioner. However, despite recent theoretical advances, our understanding of how noise influences signal propagation in deep neural networks remains limited. By extending recent work based on mean field theory, we develop a new framework for signal propagation in stochastic regularised neural networks. Our noisy signal propagation theory can incorporate several common noise distributions, including additive and multiplicative Gaussian noise as well as dropout. We use this framework to investigate initialisation strategies for noisy ReLU networks. We show that no critical initialisation strategy exists using additive noise, with signal propagation exploding regardless of the selected noise distribution. For multiplicative noise (e.g. dropout), we identify alternative critical initialisation strategies that depend on the second moment of the noise distribution. Simulations and experiments on real-world data confirm that our proposed initialisation is able to stably propagate signals in deep networks, while using an initialisation disregarding noise fails to do so. Furthermore, we analyse correlation dynamics between inputs. Stronger noise regularisation is shown to reduce the depth to which discriminatory information about the inputs to a noisy ReLU network is able to propagate, even when initialised at criticality. We support our theoretical predictions for these trainable depths with simulations, as well as with experiments on MNIST and CIFAR-10. ‡

This paper analyzes signal propagation in vanilla fully-connected neural networks in the presence of noise. For ReLU networks, it concludes that the initial weight variance should be adjusted from the "He" initialization to account for the noise scale. Various empirical simulations corroborate this claim.  Generally speaking, I believe studying signal propagation in random neural networks is a powerful way to build better initialization schemes and examining signal propagation in the presence of noise is an interesting direction. The paper is well-written and easy to read. However, I found the analysis and conclusions of this work to be rather weak and I do not believe it is appropriate for publication at NIPS at this time.  The technical contributions of this work are readily derived from eqn. (5), which is a fairly simple modification of the recursion relation studied in [1,2]. From eqn. (5), it is easy to see that multiplicative noise can be absorbed into \sigma_w, and additive noise can be absorbed into \sigma_b. While this is a useful observation, to me it seems fairly trivial from the technical perspective and insufficient to serve as the backbone of the entire paper.  Perhaps more importantly, the paper focuses entirely on "q", i.e. pre-activation variances, and does not analyze the propagation of "c", i.e. pre-activation correlations. The latter is necessary for understanding the phase diagram, the stability of fixed points, and the trainable depth scales. Even if the phase diagram ends up looking identical to the noise-free case (modulo redefinitions of \sigma_w and \sigma_b to account for the noise), this point should have been discussed and investigated.  The trainability experiments in Figure 4 go out to depth 200. While this is certainly quite deep, it may not be deep enough to demonstrate the breakdown of trainability arising from ill-conditioned Jacobians. Indeed, in [3], critically-initialized networks with ill-conditioned Jacobians can train out to depth 200, albeit much more slowly than their well-conditioned counterparts.  Overall, while I like the general idea behind this paper, I do not find the results significant enough to merit publication at NIPS at this stage.  [1] Poole, Ben, et al. "Exponential expressivity in deep neural networks through transient chaos." Advances in neural information processing systems. 2016. [2] Schoenholz, Samuel S., et al. "Deep Information Propagation." ICLR 2017. [3] Pennington, Jeffrey, Samuel Schoenholz, and Surya Ganguli. "Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice." Advances in neural information processing systems. 2017.  Edit: After reviewing the authors' response and examining the extensive new experiments, I have increased my score to 7 and vote for acceptance. The authors added new analysis on the phase diagram, convergence rates of correlations, and many other important metrics. I believe these analyses will significantly improve the paper. I still think the technical meat of the paper is rather weak, but given the extensive new experiments I think the paper is worthy of publication.


Assessing Generative Models via Precision and Recall
Recent advances in generative modeling have led to an increased interest in the study of statistical divergences as means of model comparison. Commonly used evaluation methods, such as the Fréchet Inception Distance (FID), correlate well with the perceived quality of samples and are sensitive to mode dropping. However, these metrics are unable to distinguish between different failure cases since they only yield one-dimensional scores. We propose a novel definition of precision and recall for distributions which disentangles the divergence into two separate dimensions. The proposed notion is intuitive, retains desirable properties, and naturally leads to an efficient algorithm that can be used to evaluate generative models. We relate this notion to total variation as well as to recent evaluation metrics such as Inception Score and FID. To demonstrate the practical utility of the proposed approach we perform an empirical study on several variants of Generative Adversarial Networks and Variational Autoencoders. In an extensive set of experiments we show that the proposed metric is able to disentangle the quality of generated samples from the coverage of the target distribution.

This paper contributes some original thinkings on how to assess the quality of a generative model. The new evaluation metric, as defined by the distributional precision and recall statistics (PRD), overcomes a major drawback from prior-arts: that evaluation scores are almost exclusively scalar metrics. The author(s) attributes intuitive explanations to this new metric and experimentally reached a conclusion that it is able to disentangle the quality from the coverage, two critical aspects wrt the quality of a learned synthetic sampler. An efficient algorithm is also described and theoretically justified. The quality of this work seems okay, yet I am prone to a neutral-to-negative rating. This is mainly because I am not sure I fully understand the technical details from the paper after a few attempts, and I have doubts about its value to the future development of improved generative models. I will reconsider my decision if the author(s) can properly address my concerns detailed below.   * While I strongly concur with the fact that a comprehensive set of indicators are needed to properly evaluate the quality of a generative model, it is not so obvious to me why a curve-type metric like PRD proposed in the current study is more preferable. Such metrics can be more open to (subjective) interpretations, which I found undesirable. Because this might add to the confusion of what makes a good generative model.   * Overload of symbol \bar{\apha} and \bar{\beta}.   * When doing a comparison involving more than 3 distributions, shouldn't we use the pooled samples from all the models (and data) to construct the local bins? (As oppose to pair-wise matching for the evaluation of PRD score, which I assume is what the author(s) did in the current experiments).   * The author(s) kind of implies there is this trade-off between precision and recall, at least in practice. I need more proof on this point.   * As also mentioned by the author(s), KNN may not be a reliable tool as the cluster assignment might be arbitrary.   * Fig. 7 left, adding marginal histograms might serve to better interpret the observation.   * The experimental results in the main text seem a little seem somewhat weak, as all experiments are only based on simple binary/gray-scale images. More extensive set of experiments, like some representative toy models and more sophisticated image datasets, should be considered in the main text.   * This paper does not provide any insights on how to improve generative modeling wrt the novel metric discussed. And this is a major drawback.


Multiple-Step Greedy Policies in Online and Approximate Reinforcement Learning
Multiple-step lookahead policies have demonstrated high empirical competence in Reinforcement Learning, via the use of Monte Carlo Tree Search or Model Predictive Control. In a recent work [5], multiple-step greedy policies and their use in vanilla Policy Iteration algorithms were proposed and analyzed. In this work, we study multiple-step greedy algorithms in more practical setups. We begin by highlighting a counter-intuitive difficulty, arising with soft-policy updates: even in the absence of approximations, and contrary to the 1-step-greedy case, monotonic policy improvement is not guaranteed unless the update stepsize is sufficiently large. Taking particular care about this difficulty, we formulate and analyze online and approximate algorithms that use such a multi-step greedy operator.

Summary: The paper studies multi-step greedy policies and their use in approximate policy iteration. The authors first show a negative result that soft-policy updates using the multi-step greedy policies do not guarantee policy improvement. Then the authors proposed an algorithm that uses cautious soft updates (only update to the kappa greedy policy only when assured to improve, otherwise stay with one-step greedy policy) and show that it converges to the optimal policy. Lastly the authors studied hard updates by extending APIs to multi-step greedy policy setting.   Comments:    1. Theorem 2 presents an interesting and surprising result. Though the authors presented the example in the proof sketch, but I wonder if the authors could provide more intuitions behind this? Based on the theorem, for multi-step greedy policy, it seems that h needs to be bigger than 2. So I suspect that h = 2 will still work (meaning there could exist small alpha)? Obviously h = 1 works, but then why when h = 3, the soft-update suddenly stops working unless alpha is exactly equal to 1?  I would expect that one would require larger alpha when h gets larger.   2. Alg 1 and theorem 4 are nice. While I didn’t check the proof of theorem 4, how fast the algorithm will converge compare to a simple one-step greedy version? Namely what is the exact benefit of introducing the extra q_{\kappa} here (lemma 3 shows that the contraction is still gamma, and has nothing to do with kappa here)?   3. Regarding the hard updates, how realistic is the assumption on the existence of a kappa-greedy policy oracle? This is different from a greedy oracle that one would typically use in, for instance, CPI, where the oracle can be implemented by classic cost-sensitive classification with cost-to-go evaluated simply by roll-outs. To compute such a kappa-greedy policy here, we essentially need to solve a MDP (i.e. planning), though it is reshaped and the discount factor is decreased. Using a conservative policy iteration for such a greedy oracle is strange. Why not just using CPI to solve the original problem? Can we show that using CPI as the oracle for computing kappa-greedy policy eventually resulting a faster algorithm than CPI itself, if one set kappa optimally?  Similarly, can we show that using CPI as the kappa greedy policy oracle in PSDP resulting a faster algorithm than PSDP?  After rebuttal:   Thanks for detailed comments in the rebuttal. Though understanding the limitation of kappa-greedy oracle and how to set kappa are important for practical applications of the proposed approaches, I do think the paper already contains significant contribution. I've updated my score.


Graphical Generative Adversarial Networks
We propose Graphical Generative Adversarial Networks (Graphical-GAN) to model structured data. Graphical-GAN conjoins the power of Bayesian networks on compactly representing the dependency structures among random variables and that of generative adversarial networks on learning expressive dependency functions. We introduce a structured recognition model to infer the posterior distribution of latent variables given observations. We generalize the Expectation Propagation (EP) algorithm to learn the generative model and recognition model jointly. Finally, we present two important instances of Graphical-GAN, i.e. Gaussian Mixture GAN (GMGAN) and State Space GAN (SSGAN), which can successfully learn the discrete and temporal structures on visual datasets, respectively.

This paper proposes Graphical-GAN, a variant of GAN that combines the expressivity of Graphical Models (in particular, Bayesian nets) with the generative inductive bias of Generative Adversarial Networks. Using adversarial learning, the resulting hybrid generative model minimizes for a f-divergence between the joint distributions of the generative model p(X, Z) and an auxiliary encoding distribution q(X,Z). For highly structured latent variables, such as the ones considered in this work, the discriminator's task of distinguishing X,Z samples from the two distributions can be different.   As a second major contribution, the work proposes a learning procedure inspired by Expectation Propogation (EP). Here, the factorization structure of the graphical model is explicitly exploited to make the task of the discriminator "easier" by comparing only subsets of variables. Finally, the authors perform experiments for controlled generation using a GAN model with a mixture of Gaussians prior, and a State-Space structure to empirically validate their approach.  The paper is clearly written, easy-to-follow and achieves a good balance of intuition and formalism. The paper is closely related to prior works at the intersection of inference in GANs, in particular ALI and BiGAN. However, the key novelty of the work is the use of the formalism of graphical models for structured learning and inference. That is, instead of assuming a simple dependence of Z->X, the work looks at richer, alternate factorizations which can explicitly encode prior knowledge, such as a Gausssian Mixture Model.   To the best of my knowledge, the work is technically correct. The empirical evaluation is moderately convincing (shortcomings explained in more detail later in the review). At a broader level, this work asks the following question: Does the structure of graphical models have a role to play in generative modeling? The answer is a clear yes to me, and I was highly pleased to see this work taking a step in this direction. As the authors have noted, the first work in this regime is due to Johnson et al. (done in the context of VAEs) --- however this work is the first to do for adversarially learned generative models.  For the rebuttal phase, I would like to hear the author's responses to the following questions/comments:  1. Since we are using GANs, likelihood evaluation is intractable (except for invertible decoders). Then, the key benefit of Graphical-GANs over other models is controlled generation and inferring latent representations in the presence of prior information (such as the MNIST dataset can be assumed to be generated from 10 independent GANs). InfoGAN enjoys all of these properties. In L199, it is said that GMGAN doesn't need a regularization loss and is heirarchical, which makes it superior. Even though these distinctions are valid, I don't see why the Graphical-GAN approach should be preferred over InfoGAN, both in theory and experiments.   In fact, Graphical-GNN requires additional information tn the form of a strcutured graphical model. On the other hand, InfoGAN seems to automatically uncover this hidden structure and hence, would be more immune to "structure mismatches" between the model and the assumed data. Can the authors comment on this and if possible, report summary experimental results comparing with InfoGAN?  2.  Missing experiments -- Why are the entries in first and third column of Table 1 missing? What about clustering accuracy on CIFAR-10 for the different baselines?  3. The use of the term generalization in L275 is misleading. The only widely accepted definition of generalization I am familiar with posits that the train and test performance of a learned model should be similar to claim that the model is generalizing. This is really the opposite of generalization we see in Figure 8 and lines 275-277 where the training dataset is very different from the generated data in the length of the frames.  4. Looking at the full page of miniature samples in Figure 1 and 2 in Appendix E and Figure 5 in Appendix F tells me nothing about the performance of GMGAN-L on axis of interest. Since these are all labelled/attributed datasets, I would like to see some quantitative metrics such as average clustering accuracy, inter-intra clustering distances etc. to be able to derive a conclusion.  ---- ---- Thanks for the response. I am convinced with the alternate explanation that the proposed framework can solve a diversity of tasks with much richer graphical structures, unlike InfoGAN. This reduces the significance of the MNIST experiments compared to prior work. And R2 has raised some valid concerns with the SSGAN experiments that the authors could address in a later version.


Flexible and accurate inference and learning for deep generative models
We introduce a new approach to learning in hierarchical latent-variable generative models called the "distributed distributional code Helmholtz machine", which emphasises flexibility and accuracy in the inferential process. Like the original Helmholtz machine and later variational autoencoder algorithms (but unlike adversarial methods) our approach learns an explicit inference or "recognition" model to approximate the posterior distribution over the latent variables. Unlike these earlier methods, it employs a posterior representation that is not limited to a narrow tractable parametrised form (nor is it represented by samples). To train the generative and recognition models we develop an extended wake-sleep algorithm inspired by the original Helmholtz machine. This makes it possible to learn hierarchical latent models with both discrete and continuous variables, where an accurate posterior representation is essential. We demonstrate that the new algorithm outperforms current state-of-the-art methods on synthetic, natural image patch and the MNIST data sets.

The paper describes a novel approach to perform posterior inference and learning in deep generative models. This work is mainly motivated by the fact that recent VAE approaches still have difficulties in learning models with multiple layers, and that they have difficulties with adequately capturing structured posteriors. The key contributions in this paper are  * a novel approach to capture the posterior indirectly, using an inference network predicting expectations of random statistics w.r.t. the true posterior, and * demonstrating how to exploit this indirect representation for learning the generative model. To that end, the authors propose the Distributed Distributional Code (DDC) Helmholtz Machine, an extension to the classical Helmholtz Machine, including an extended wake-sleep algorithm.  Strengths: + a quite refreshing approach to approximate inference and learning. The proposed approach combines several existing techniques in an intriguing way. In particular, I think that the approach to represent the posterior via estimated expectations of random statistics is a very powerful idea, with a potentially large impact on approximate inference/learning. Also the approach to circumvent an explicit representation of the approximate posterior density, but rather to derive required expectations (gradients) in a direct way, is a promising direction. + Experimental evaluations demonstrate -- to a certain extent -- the efficacy of the approach, with partially dramatic improvements over variational approaches.  Weaknesses: - the approach inherits the disadvantages of wake-sleep. Thus, there is no clear objective optimized here and the method might even diverge. Furthermore, there is an additional approximation step which was not present in the original wake-sleep algorithm, namely the prediction of the required expectations/gradients. While the authors mention that the original wake-sleep algorithm has these drawbacks, they don't discuss them at all in their approach. - the motivation to fix VAEs' problem to learn deep models is addressed rather briefly in the experiments, by learning a model over binarized MNIST and 3 hidden sigmoid layers. This is not too convincing.  Quality: The paper is technically sound, in that way that the choices made are clear and very convincing. However, as the approach is based in wake-sleep, it is clear that the approach is heuristic, in particular as it includes a further approximation (expectations/gradients used for learning). The consequences of this fact and weaknesses of the model should be discussed in a more open way and ideally be addressed in the experimental evaluation (e.g. showing fail cases, or discussing why there are none).   Clarity: The paper is very well written and easy to follow.  Originality: The approach presented is, to the best of my knowledge, novel, natural and elegant.   Significance: Although the proposed is heuristic, it has a high potential to trigger further research in this direction.  Summary: while the paper has its weaknesses (heuristic, which should be better discussed), its originality and potential inspiration for further work in this direction are the main reasons why I recommend an accept.   *** EDIT *** I read the authors' reply and I'm rather happy with it.  While their point is convincing that the approximation in DDC-HM can be made arbitrarily close, I'm still not sure if we can guarantee stable training. The authors claim that it can be shown that the algorithm approaches the vicinity of a stationary point of the likelihood, if the gradient approximation error is small. I think this might be a crucial point, but I don't fully understand if this implies convergences (rather than oscillatory behavior) and how small the error needs to get. In any case, the authors should include this result in their paper.  And again, nice and refreshing work, I stick with my original rating.


An Efficient Pruning Algorithm for Robust Isotonic Regression
We study a generalization of the classic isotonic regression problem where we allow separable nonconvex objective functions, focusing on the case where the functions are estimators used in robust regression. One can solve this problem to within -accuracy (of the global minimum) in O(n/ ) using a simple dynamic program, and the complexity of this approach is independent of the underlying functions. We introduce an algorithm that combines techniques from the convex case with branchand-bound ideas that is able to exploit the shape of the functions. Our algorithm achieves the best known bounds for both the convex case (O(n log(1/ ))) and the general nonconvex case. Experiments show that this algorithm can perform much faster than the dynamic programming approach on robust estimators, especially as the desired accuracy increases.

This paper studies fast algorithms for isotonic regression under general loss functions. The author(s) build upon an existing dynamic programming (DP) approach by suggesting two main strategies to prune points -- one based on pruning with lower/upper bounds based on the Lipschitz constants of the loss functions, and another based on "linear underestimators" (which are assumed to be computable in constant time).  Overall this is a well written paper. There are minor language issues and ommisions which can be ironed. A few examples are  line 69 what is S in argmin_{z in [0,1]} \sum_{i \in S} f_i(z) line 181 the tighter "the" underestimator .. line 240 "and run" should be "and ran"  In my opinion the proofs presented in the paper appear a bit heuristic and "hand-wavy", so one suggestion that I have is to strengthen the proofs.   One thing that does not become very clear from the paper is the following: does the DP have complexity O(nk) in the general case, and O(n log k) in the convex case (or only a special variant of it has O(n log k) in the convex case)? If so, then the improvement of this paper is not in improving those orders, but is in improving the constants of the orders?  Another related question is does the DP require the loss functions to be Lipschitz, as required by the pruning algorithm? If not that would be a disadvantage to the pruning algorithm which seems to rely on the Lipschitz assumptions (and is hence not applicable to the Euclidean distance which is the classic isotonic regression).   Finally, I think that including back the omitted plot "running time vs k" would be beneficial since one would see how much real time it takes to run the algorithm.