Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks
High sensitivity of neural networks against malicious perturbations on inputs causes security concerns. To take a steady step towards robust classifiers, we aim to create neural network models provably defended from perturbations. Prior certification work requires strong assumptions on network structures and massive computational costs, and thus the range of their applications was limited. From the relationship between the Lipschitz constants and prediction margins, we present a computationally efficient calculation technique to lower-bound the size of adversarial perturbations that can deceive networks, and that is widely applicable to various complicated networks. Moreover, we propose an efficient training procedure that robustifies networks and significantly improves the provably guarded areas around data points. In experimental evaluations, our method showed its ability to provide a non-trivial guarantee and enhance robustness for even large networks.

This paper proposes a computationally efficient calculation technique that lower-bounds the size of adversarial perturbations that can deceive networks, and an effective training procedure, which robustifies networks and significantly improves the provably guarded areas around data points. The contribution of this paper is proposing an intuitive way to measure the slope to calculate the upper-bounds of gradient and provide a widely available and highly scalable method that ensures large guarded areas for a wide range of network structures. There are certain contribution and originality in the literature.  Here I am concerned with the following two questions: 1.    This paper defines a guarded area for a network F and a data point X as a hypersphere with a radius c. This definition is so strong that may conflict with many real world applications. 2.    The authors transfer the problem into solving the Lipschitz constant and prove bounds in diffident functions and do some experiments to evaluate the performance of their method. But lack of the comparative experiments with some existing methods that make the experiment unconvincing.


Adversarial Multiple Source Domain Adaptation
While domain adaptation has been actively researched, most algorithms focus on the single-source-single-target adaptation setting. In this paper we propose new generalization bounds and algorithms under both classification and regression settings for unsupervised multiple source domain adaptation. Our theoretical analysis naturally leads to an efficient learning strategy using adversarial neural networks: we show how to interpret it as learning feature representations that are invariant to the multiple domain shifts while still being discriminative for the learning task. To this end, we propose multisource domain adversarial networks (MDAN) that approach domain adaptation by optimizing task-adaptive generalization bounds. To demonstrate the effectiveness of MDAN, we conduct extensive experiments showing superior adaptation performance on both classification and regression problems: sentiment analysis, digit classification, and vehicle counting.

Quality: The paper tackles the problem of domain adaptation. The main proposal of the paper is to find a multi-source generalization bound equivalent of a previously established single-source bound in [9]. Accordingly, two objective functions are devised to tighten the bound for multi-source domain adaptation. The experiments clearly shows that optimizing the proposed objective is more effective than naive extension of single-source objectives. The main paper is technically sound both the theories and the experiments seem valid to the eye of the reviewer.   Clarity: The paper is easy to follow. It motivates the problem, puts forward the formal notation in a concise and readable format and builds up the proposed approach on that.  The hyper-parameters necessary to reproduce the experimental results come in the appendix. The paper would strongly benefit from a thorough theoretical and empirical analysis of the convex coefficients \alpha on the tightness of the bound as well as the optimization. The reviewer appreciates the design of intuitive adaptive coefficients in 6 but more discussions and analysis would have been very helpful.  Originality: The paper builds on top of an existing work on single source domain adaptation and generalizes it to a special case of multi-source domain adaptation. The assumption about the setups of the multi-source domain adaptation (availability of mk unlabelled target data) is not too restrictive.  The related works section is limited, other works on multi-source domain adaptation exist that are not cited. For instance, Sun et al. NIPS 2011, or Choi et al. CVPR 2018.   Significance: - the paper puts forward a specific generalization of a previously published single-source domain adaptation error bound to multiple sources. The generalization is simple and intuitive.  - The paper tests that optimizing the derived bounds is better than naive approaches to multi-source domain adaptation. In that regard, it conclusively shows the benefits. However, it fails to compare to other approaches that are taken for multi-source domain adaptation (such as the ones mentioned above). - Another issue with the experiments is that when comparing C-DANN with the proposed MDAN it seems like C-DANN can achieve comparable results and on rare occasions even better results. This raises the need for baselines where more advanced domain adaptation techniques are used with combined source domains as a single domain.  ------------------- I read the rebuttal and the reviewers' comments. It seems I agree with all the reviewers' comments and the rebuttal makes sense. I will keep my rating and vouch for the acceptance of the paper.


Distributed Multitask Reinforcement Learning with Quadratic Convergence
Multitask reinforcement learning (MTRL) suffers from scalability issues when the number of tasks or trajectories grows large. The main reason behind this drawback is the reliance on centeralised solutions. Recent methods exploited the connection between MTRL and general consensus to propose scalable solutions. These methods, however, suffer from two drawbacks. First, they rely on predefined objectives, and, second, exhibit linear convergence guarantees. In this paper, we improve over state-of-the-art by deriving multitask reinforcement learning from a variational inference perspective. We then propose a novel distributed solver for MTRL with quadratic convergence guarantees.

Summary: This paper presents a distributed Multitask Reinforcement Learning (MTRL) from a variational inference perspective. It improves the state-of-art ADMM by a distributed solver using Chebyshev polynomials with \textit{quadratic convergence guarantees}. The theoretical analysis is verified with extensive experiments, but to strengthen the result, further experiments on real large-scale Multitask Reinforcement Learning problems are needed.  Quality: Technically, this paper is rather strong. First, the MTRL is formulated as a variational inference problem by introducing a variational trajectory distribution to arrive at the maximization of variational lower bound. Variational distribution and policy parameters (shared and task-specific) could be optimized alternatively. Furthermore, to learn the shared parameter, the authors incorporate Laplacian matrices from Graph structure into the objective function, with a consensus constraint for the shared parameter to share and transfer knowledge. To solve the scalability issues, the authors adopt Chebyshev polynomials and de-centralize, distribute the computations of Newton direction, and prove the quadratic convergence rate.  The experiments' scales are pretty small, even for the larger random network with only 150 nodes and 250 edges. It is better to study the really large-scale problems and investigate the speedup w.r.t computation units e.g. the number of CPU/GPU.  Clarity: The majority of this paper is clear and well written and organized. However, there are some minor issues. -- Notation inconsistency: for action, line 57, 58, 67 use $a$, while line 103 uses $u$; for $q$, line 208 denotes dual function $q(\lambda)$, yet line 76 denotes variational distribution $q_\phi(\tau)$. -- Figures: Fig. 2(a) is too small, it might be better to use log-log plot for Fig. 6 -- Algorithm: it might be better to move algorithm 1 into the main paper, instead of the Appendix. -- Experiment: As communication overhead is not ignorable, it is better to show the comparison of overall time (communication + running time). -- Computing Architecture: It is better to illustrate how master and worker nodes are organized to distribute the Newton direction computation. This will help us understand the distribution algorithm and communication overhead. -- Results: In Fig. 3(b) for CP, SDD-Newton does not seem to be the best. In Fig. 6(c), distributed SDD Newton is no better than Distributed ADMM for the final objective values.  Originality: The major contribution of this paper is to adopt Chebyshev polynomials and distribute the computing of Newton search direction with a proof of quadratic convergence rate. It is natural to use Chebyshev polynomials to solve the system of linear equations described via diagonally dominant matrices, where the Newton search direction is such kind of problem. However, the distributed Chebyshev Solver is non-trivial and it is an interesting idea to introduce such a technique into distributed Multitask Reinforcement Learning.  Significance: Scalability is an important issue in Multitask Reinforcement Learning and it is the right direction to distribute the computation to as many machines as possible. Thus, the quadratic convergence guarantees an important result for the distributed Multitask Reinforcement Learning. However, the trade-off between communication and computation is the next important issue needed to be seriously considered.  ------- After Rebuttal --------- I have checked the others' reviews. Most of my concerns are addressed by the author feedback.


Dimensionality Reduction for Stationary Time Series via Stochastic Nonconvex Optimization
Stochastic optimization naturally arises in machine learning. Efficient algorithms with provable guarantees, however, are still largely missing, when the objective function is nonconvex and the data points are dependent. This paper studies this fundamental challenge through a streaming PCA problem for stationary time series data. Specifically, our goal is to estimate the principle component of time series data with respect to the covariance matrix of the stationary distribution. Computationally, we propose a variant of Oja's algorithm combined with downsampling to control the bias of the stochastic gradient caused by the data dependency. Theoretically, we quantify the uncertainty of our proposed stochastic algorithm based on diffusion approximations. This allows us to prove the asymptotic rate of convergence and further implies near optimal asymptotic sample complexity. Numerical experiments are provided to support our analysis.

Summary: The paper consider the setting of streaming PCA for time series data which contains two challenging ingredients: data stream dependence and a non-convex optimization manifold. The authors address this setting via downsampled version of Oja's algorithm. By closely inspecting the optimization manifold and using tools from the theory of stochastic differential equations, the authors provide a rather detailed analysis of the convergence behavior, along with confirming experiments on synthetic and real data.  Evaluation: Streaming PCA is a fundamental setting in a topic which becomes increasingly important for the ML community, namely, time series analysis. Both data dependence and non-convex optimization are still at their anecdotal preliminary stage, and the algorithm and the analysis provided in the paper form an  interesting contribution in this respect. The paper is well-structured, although I find some key definitions missing, and some technical parts somewhat hard-to-follow.   Comments: L47 - I find 'Moreover, theses results..' somewhat vague. Can you further elaborate on the nature of the uncertainty not captured by current convergence analyses from your point of view? L115 - Many terms here are used without providing a definition first. E.g., $\mathcal{F}$ in $(\mathcal{S},\mathcal{f})$, positive recurrent, transition kernel,.. L162 - 'The eigengaps in ... is identfible' seems to have a grammatical mistake. L169 - Can you put your statements regarding the taxonomy of stationary points on a more rigorous ground? L274 - Most of the proof sketch addresses the continuous analog of the discrete dynamic, i.e., (stochastic) gradient flow. The derivation of the convergence rate for the original discrete setting is less clear to me. Also, I would suggest adding an explicit statement covering the conditions and convergence rates implied by the analysis, at earlier stages of the paper before the proof sketch (as well as other contributions made by the paper - such as the intimate analysis of the algorithm at different stage of the optimization process using SDE).


Maximizing acquisition functions for Bayesian optimization
Bayesian optimization is a sample-efficient approach to global optimization that relies on theoretically motivated value heuristics (acquisition functions) to guide its search process. Fully maximizing acquisition functions produces the Bayes' decision rule, but this ideal is difficult to achieve since these functions are frequently non-trivial to optimize. This statement is especially true when evaluating queries in parallel, where acquisition functions are routinely non-convex, highdimensional, and intractable. We first show that acquisition functions estimated via Monte Carlo integration are consistently amenable to gradient-based optimization. Subsequently, we identify a common family of acquisition functions, including EI and UCB, whose properties not only facilitate but justify use of greedy approaches for their maximization.

Summary: This paper proposes a new way of solving the problem of optimizing the acquisition function in Bayesian optimization. The idea is to look at most common acquisitions as the optimization of an expectation of a function, which can be reformulated using the 'reparametrization trick' and optimized using Monte Carlo. The papers shows the reformulation of various acquisitions and presents some interesting experimental results.  Strengths and weaknesses of the submission: The paper is really well written and it attacks a problem that affects seriously the performance of new Bayesian optimization methods. The approach is very original and well illustrated and I think that this approach has the potential to become a standard practice in Bayesian optimization alternatively to current approaches that are not even well formalized (each BayesOpt library does it it in its own way). In my opinion this paper brings some foundational aspects to a part of the the optimization loop that has been ignored by the community, in general.  Quality: this is a high quality work both because the idea is new but also because the paper is very nicely written.  Clarity: the paper is very clear   Originality: The method to reparametrize the acquisition is know but its application to this domain is new.  Significance: the method is significant for the BayesOpt community in particular and for the ML community in general (given that these techniques are becoming part of the standard toolkit of the ML practitioner).


A dual framework for low-rank tensor completion
One of the popular approaches for low-rank tensor completion is to use the latent trace norm regularization. However, most existing works in this direction learn a sparse combination of tensors. In this work, we fill this gap by proposing a variant of the latent trace norm that helps in learning a non-sparse combination of tensors. We develop a dual framework for solving the low-rank tensor completion problem. We first show a novel characterization of the dual solution space with an interesting factorization of the optimal solution. Overall, the optimal solution is shown to lie on a Cartesian product of Riemannian manifolds. Furthermore, we exploit the versatile Riemannian optimization framework for proposing computationally efficient trust region algorithm. The experiments illustrate the efficacy of the proposed algorithm on several real-world datasets across applications.

The article focus on tensor completion by trace-norm regularization. In this approach, tensor with missing data are completed with a decomposition as a sum of K low-rank tensors. The rank of each of these tensor components is not fixed but constrained by a trace-norm regularization on the specific matricized form of each tensor (so tensor component K will be constrained to have a low-rank k-mode matricization). The main contribution is to propose another regularization than latent-trace norm which tends to eliminate components, thus enabling a non-sparse decomposition.  The proposed trace norm regularizer can be rewritten through variational characterization as the solution of a minimization problem on the space of positive semi-definite matrices with unit trace (defined as the spectrahedron manifold in Journée at al./2008)  and two novel formulations whose main interests are to depends on only one dual tensor (compared to K components) and K matrices are presented. As the matrices belongs to the spectrahedron manifold, they can be parametrized through a product of a fixed-rank matrices (whose frobenius norm is 1) with any element of the orthogonal group. As a consequence,  the fixed-parameters problems should  be solved through optimization on a quotient manifold. Here a Trust-Region (second order) algorithm is chosen which necessitates the computation of the Riemannian gradient, the Riemannian hessian and a retraction operator. All elements are provided and detailed in the supplementary material.  Last, several experiments (color image completion, link prediction, movie recommendation) are presented that shows the general efficiency of the method. There are some missing details like the chosen ranks of the parametrization of the spectrahedron matrices which could influence the effectiveness of the method. However as codes are provided, the reproductivity of some of the  experiments is possible.  My main complaint with this work is: as this works focus in tensor completion, it eschews to bring into the light its relationship with matrix completion methods. In particular, there are a lot of elements in common with the recent (ICML’18) work  [P. Jawanpuria and B. Mishra. A Unified framework for structured low-rank matrix learning. International Conference on Machine Learning, 2018.] (cited in the text as [17]). In particular, the dual trick, the main Riemannian components necessary for the Trust Region implementation (projections, retraction) are the same.  Another remark is that the use of trace-norm regularization enables to not specify the ranks of each components of the decomposition, whereas the chosen parametrization of the elements of the spectrahedron manifold makes the choice of ranks of the matrix again necessary.   A summary is that the addressed problem is interesting, the algorithms proposed make a good use of the Riemannian framework and the experiments are good. The main negative point is that the contributions of this work may rely a lot to similar work on matricial decomposition


Modern Neural Networks Generalize on Small Data Sets
In this paper, we use a linear program to empirically decompose fitted neural networks into ensembles of low-bias sub-networks. We show that these sub-networks are relatively uncorrelated which leads to an internal regularization process, very much like a random forest, which can explain why a neural network is surprisingly resistant to overfitting. We then demonstrate this in practice by applying large neural networks, with hundreds of parameters per training observation, to a collection of 116 real-world data sets from the UCI Machine Learning Repository. This collection of data sets contains a much smaller number of training examples than the types of image classification tasks generally studied in the deep learning literature, as well as non-trivial label noise. We show that even in this setting deep neural nets are capable of achieving superior classification accuracy without overfitting.

This paper presents an interesting idea, which is that deep neural networks are able to maintain reasonable generalization performance, even on relatively small datasets, because they can be viewed as an ensemble of uncorrelated sub-networks.    Quality:  The decomposition method seems reasonable, except for the requirement for the model and the sub-nets to achieve 100% training accuracy.  While there are some datasets where this will be reasonable (often high-dimensional datasets), there are others where such an approach would work very badly.  That seems to me a fundamental weakness of the approach, especially if there are datasets of that nature where deep neural nets still perform reasonably well.  For a random forest, we have an unweighted combination of base classifiers, but it is a learned combination in the case of the decomposed sub-networks, and the weights are tuned on the training data.  Unless each component has similar weights it is not clear to me the comparison with RF is truly apposite.  The top hat dataset in figure 1 does not seem greatly convincing as the generalization of RF and the deep neural network looks quite poor, both have over-fitted the training data quite badly (hard to see how it could have over-fitted the data any more that it has done).  It isn't clear how e.g. categorical attributes have been coded.  This should be set out clearly so the work is replicable.  The simplicity of the experimental set-up, however, with the key parameters stated, is a nice aspect of the paper.  I'm not sure I agree with the description of the results in lines 224-239.  In both cases, the average difference in performance seems quite large, and indeed the deep neural networks are statistically inferior to the RF, suggesting they have not performed all that well, although it is a good observation that their performance has not completely collapsed, that seems a rather weak finding.  The thing that seems missing to me is a discussion of whether the correlation in the sub-nets is correlated with their generalisation performance.  While there is a plot of the error correlation in Fig 4 (b) there doesn't seem to be a way of relating that to generalisation performance.  Perhaps re-order the datasets according to the ratio of their performance relative to RF?  It is good to see deep learning applied to smaller datasets, rather than the image datasets typically used as benchmarks, it is just as interesting to find out what doesn't work, as well as what does.  Clarity:  The paper is quite well written, I had no problems understanding it (I hope!).  Rao should be capitalised in reference 16.  Originality:  The idea seems reasonable and novel, I don't recall having seen a similar decomposition suggested.  Significance:  I think this paper is interesting.  I have found MLP networks with large hidden layer sizes often perform reasonably well on small datasets, and it would be useful to have some insight into why this happens (although with a single hidden layer, the decomposition into de-correlated sub-networks seems less applicable).  In my experience deep neural networks do not work as well as "shallow" neural networks for this type of data, but even for datasets where there are better algorithms, such insight is still useful.


BinGAN: Learning Compact Binary Descriptors with a Regularized GAN
In this paper, we propose a novel regularization method for Generative Adversarial Networks, which allows the model to learn discriminative yet compact binary representations of image patches (image descriptors). We employ the dimensionality reduction that takes place in the intermediate layers of the discriminator network and train binarized low-dimensional representation of the penultimate layer to mimic the distribution of the higher-dimensional preceding layers. To achieve this, we introduce two loss terms that aim at: (i) reducing the correlation between the dimensions of the binarized low-dimensional representation of the penultimate layer (i.e. maximizing joint entropy) and (ii) propagating the relations between the dimensions in the high-dimensional space to the low-dimensional space. We evaluate the resulting binary image descriptors on two challenging applications, image matching and retrieval, and achieve state-of-the-art results.

Summary This paper proposes a variant of GAN to learn compact binary descriptors for image patch matching. The authors introduce two novel regularizers to propagate Hamming distance between two layers in the discriminator and encourage the diversity of learned descriptors. The presentation is easy to follow, and the method is validated by benchmark datasets.   Major concerns: [Motivation and Presentation] First of all, it’s not so clear the reason why adversarial training helps to learn compact binary descriptors. In addition, the motivation on DMR is also not fully addressed in my sense. In my understanding, the discriminator has two binary representation layers; one of them has the larger number of bits and the other is used for the compact binary descriptor. However, it’s not clearly described in the current draft.  [About Experiments] There exists a simple baseline by just combining state-of-the-art image descriptors with binary embedding algorithms. Specifically, in stead of learning descriptors and its binary representation together, let us split two parts and use SOTA algorithms in both directions. For example, what about using the following algorithm followed by deep hashing to generate compact binary descriptors? - Image Patch Matching Using Convolutional Descriptors with Euclidean Distance, ACCV’16 workshop, https://arxiv.org/abs/1710.11359 I believe that there exist more follow-up works to learn more meaningful descriptors for matching image patches.  Preliminary rating: borderline This paper has some potential to be a good application of GAN, which learns compact binary descriptors for image matching. Yet, the motivation on network architecture and experiments are not sufficient. Please elaborate them in the rebuttal, if possible.  == Updated review == The rebuttal addressed my concerns on the experiment results. Yet, I still think that the technical novelty on the proposed methods is not strong for NIPS. Specifically, the entropy regularizer has been studied to learn decorrelated hash bits before;  - SPEC Hashing: Similarity Preserving algorithm for Entropy-based Coding, CVPR’10. Though the exact form used in this manuscript might not been employed before, it is quite common to learn decorrelated bits by maximizing entropy on them.   Moreover, the other regularizer (DMR) is designed to reduce the dimensionality of high-dimensional binary codes learned from an intermediate layer of discriminator. It’s a very natural and practical regularizer, but I’m not sure that it’s solely enough to be a nips paper. Therefore, I haven’t changed my original score (5).


Generalizing Point Embeddings using the Wasserstein Space of Elliptical Distributions
Embedding complex objects as vectors in low dimensional spaces is a longstanding problem in machine learning. We propose in this work an extension of that approach, which consists in embedding objects as elliptical probability distributions, namely distributions whose densities have elliptical level sets. We endow these measures with the 2-Wasserstein metric, with two important benefits: (i) For such measures, the squared 2-Wasserstein metric has a closed form, equal to a weighted sum of the squared Euclidean distance between means and the squared Bures metric between covariance matrices. The latter is a Riemannian metric between positive semi-definite matrices, which turns out to be Euclidean on a suitable factor representation of such matrices, which is valid on the entire geodesic between these matrices. (ii) The 2-Wasserstein distance boils down to the usual Euclidean metric when comparing Diracs, and therefore provides a natural framework to extend point embeddings. We show that for these reasons Wasserstein elliptical embeddings are more intuitive and yield tools that are better behaved numerically than the alternative choice of Gaussian embeddings with the Kullback-Leibler divergence. In particular, and unlike previous work based on the KL geometry, we learn elliptical distributions that are not necessarily diagonal. We demonstrate the advantages of elliptical embeddings by using them for visualization, to compute embeddings of words, and to reflect entailment or hypernymy.
1 Introduction must comply with sophisticated desiderata. Naturally, more general and flexible approaches in which the embedding space Y needs not be Euclidean can be considered, for instance in generalized MDS on the sphere [Maron et al., 2010], on surfaces [Bronstein et al., 2006], in spaces of trees [Bȃdoiu et al., 2007, Fakcharoenphol et al., 2003 or, more recently, computed in the Poincaré hyperbolic space [Nickel and Kiela, 2017].
Probabilistic Embeddings. Our work belongs to a recent trend, pioneered by Vilnis and McCallum, who proposed to embed data points as probability measures in R d [2015], and therefore generalize point embeddings. Indeed, point embeddings can be regarded as a very particular-and degeneratecase of probabilistic embedding, in which the uncertainty is infinitely concentrated on a single point (a Dirac). Probability measures can be more spread-out, or event multimodal, and provide therefore an opportunity for additional flexibility. Naturally, such an opportunity can only be exploited by defining a metric, divergence or dot-product on the space (or a subspace thereof) of probability measures. Vilnis and McCallum proposed to embed words as Gaussians endowed either with the KullbackLeibler (KL) divergence or the expected likelihood kernel [Jebara et al., 2004]. The Kullback-Leibler and expected likelihood kernel on measures have, however, an important drawback: these geometries do not coincide with the usual Euclidean metric between point embeddings when the variances of these Gaussians collapse. Indeed, the KL divergence and the`2 distance between two Gaussians diverges to 1 or saturates when the variances of these Gaussians become small. To avoid numerical instabilities arising from this degeneracy, Vilnis and McCallum must restrict their work to diagonal covariance matrices. In a concurrent approach, Singh et al. represent words as distributions over their contexts in the optimal transport geometry [Singh et al., 2018].
Contributions. We propose in this work a new framework for probabilistic embeddings, in which point embeddings are seamlessly handled as a particular case. We consider arbitrary families of elliptical distributions, which subsume Gaussians, and also include uniform elliptical distributions, which are arguably easier to visualize because of their compact support. Our approach uses the 2-Wasserstein distance to compare elliptical distributions. The latter can handle degenerate measures, and both its value and its gradients admit closed forms [Gelbrich, 1990], either in their natural Riemannian formulation, as well as in a more amenable local Euclidean parameterization. We provide numerical tools to carry out the computation of elliptical embeddings in different scenarios, both to optimize them with respect to metric requirements (as is done in multidimensional scaling) or with respect to dot-products (as shown in our applications to word embeddings for entailment, similarity and hypernymy tasks) for which we introduce a proxy using a polarization identity.

The paper proposes using elliptical distribution (specifically uniform elliptical distributions) as probability measures for representing data instances in an embedding space. The work is inspired by prior work where the KL divergence was used for Gaussian distribution based embeddings. A motivating factor versus previous work is ensuring consistent behavior as the distributions collapse to Dirac delta functions. The papers use the 2-Wasserstein divergence/distance (motivated by its connection to optimal transport) as the metric for the embedding space. This divergence/distance has a known closed form expression for uniform elliptical distributions that relies on the Bures/Kakutani distance metric. The paper then explores manifold optimizations of the divergence in terms of the covariance matrix, and uses a factored parameterization of the covariance matrix for the optimization. To compute the gradient of the Bures distance a Newton-Schulz algorithm is used to find the matrix square roots and their inverses, which is a more parallelizable manner than the eigendecompostion. The approach is applied to a simple multidimensional scaling example and then to word embedding data sets that have previously been used with other probability distribution based embeddings.   Strength: A novel idea that is fairly well conveyed. The discussion of the manifold optimization and Figure 2 are useful. The optimization of the gradient implementation seems to be well designed. It seems like a fairly complete piece of work with the potential for further refinement and application.  Weaknesses: The text should include more direct motivation on why the added flexibility of an elliptical embedding is useful for modeling versus standard point embedding, especially relating to the hypernymy experiment.    The equation between lines 189 and 190 appears to be wrong. By Equation 2, the trace term should be divided by $d+2$, unless the covariances are pre-scaled.   The arguments (including Figure 3) for using the precision versus the covariance for visualization could be removed. The counterintuitive nature appears to be subjective and distracts from the modeling purpose of a probability distribution based embedding.  Reference for the tractable (computable) form of Bures distance should be Uhlmann. A. Uhlmann, "The “transition probability” in the state space of a∗-algebra," Rep. Math. Phys. 9, 273 (1976)  Some issues with typesetting appear to be problems with inline figures and sectioning.  Minor errors and corrections: Line 25 "are extremely small (2 or 3 numbers" -> "can be represented in two or three dimensional space.  Line 37 add comma around the phrase "or more ... $\langle y_i,y_j\rangle$" Line 55 The phrase "test time" is not clear in this context. Line 55 I cannot find any relevant comment in Table 1. Line 63 "scenarii" Line 69 Should notation for $M$ in pseudo-inverse match other cases (i.e., boldface)? Line 117 Typesetting and line number are messed up by inline figure 1. Line 130 $B$ should be $b$, and $C$ should be $B$.  Line 132 Reasoning for the counterintuitive visualizations is that for uniform distributions the divergence decreases if the scale decreases since the density is more concentrated. Line 146 "adressed" Line 152 Here and elsewhere the references appears sometimes with both parentheses and brackets. Line 155 "on the contrary" to what? This seems to be incorrectly copied from the introductory paragraph in Section 3.  Line 157 "to expensive" Line 166 "w.r.t" Line 177 Should be supplementary material or supplement.  Line 183.1 Should be "a general loss function" since $\matchal{L}$ is indicated. Line 214 "matrics" Line 215 "would require to perform" "would require one to perform" Line 233 Elliptical should be lower case Line 236 "is to use their precision matrix their covariances" ? Line 242 $y_i,y_j$ -> $y_i-y_j$ Line 243 Missing comma before $\mathbf{a}_n$ Figure 5 caption fix the initial quotation mark before Bach. Line 289 "asymetry" Line 310 "embedd"  Line 315 Should clarify if this is the Bures dot product or the Bures cosine. Line 323 "interesections"  The reference list is poorly done: lack of capitalization of proper nouns, inconsistent referencing to journal names (some with abbreviations some without, some missing title case), some URL s and ISSN inserted sporadically, and Inconsistent referencing to articles on arXiv.   ------ Author rebuttal response ----- The author rebuttal helped clarify and I have raised my score and edited the above.   The authors have promised to clarify the dot product equation that appears inconsistent to me.   I agree with Reviewer 3 that the visualization issue is not convincing and would caution elaborating on this somewhat tangential issue more. The original intuition was that the distance between a point and a uniform ellipsoid increase as the scale of the latter increases. However, with the additive nature of the similarity metric (dot product) used for the word embedding this doesn't appear to be the case. Switching the covariance with precision for the uniform elliptical embeddings seems to make it more confusing for Figure 5, where the embedding is trained using the dot product rather than distance.  I also hope the authors take care in carefully proofreading the manuscript and adhering to NIPS template for typesetting.


Deep Functional Dictionaries: Learning Consistent Semantic Structures on 3D Models from Functions
Various 3D semantic attributes such as segmentation masks, geometric features, keypoints, and materials can be encoded as per-point probe functions on 3D geometries. Given a collection of related 3D shapes, we consider how to jointly analyze such probe functions over different shapes, and how to discover common latent structures using a neural network -even in the absence of any correspondence information. Our network is trained on point cloud representations of shape geometry and associated semantic functions on that point cloud. These functions express a shared semantic understanding of the shapes but are not coordinated in any way. For example, in a segmentation task, the functions can be indicator functions of arbitrary sets of shape parts, with the particular combination involved not known to the network. Our network is able to produce a small dictionary of basis functions for each shape, a dictionary whose span includes the semantic functions provided for that shape. Even though our shapes have independent discretizations and no functional correspondences are provided, the network is able to generate latent bases, in a consistent order, that reflect the shared semantic structure among the shapes. We demonstrate the effectiveness of our technique in various segmentation and keypoint selection applications.

Main idea --------- In this work, the authors use a neural network to discover common latent structures in a given collection of related 3D shapes. They formalize the problem as finding shape-dependent dictionaries of basis functions that are meant to capture semantic information shared among all shapes. Interestingly, the aim is to discover such structures even in the absence of any correspondence information.   Strengths - Contributions ------------------------- Introduction is well written and related work seems well described.   1. The model does not require (costly) precomputed basis functions (such as Laplace-Beltrami eigenfunctions) as most approaches.  2. Dictionaries are application-driven, atoms can approximate non-continuous / non-smooth functions.  3. No need to explicitly define canonical bases to synchronize shapes, the neural network becomes the synchronizer. 4. No assumption of a universal dictionary, a data-dependent dictionary allows non-linear distortion of atoms but still preserves consistency.  The results seem to improve upon recent works.  Weaknesses ---------- The main section of the paper (section 3) seems carelessly written. Some obvious weaknesses:  - Algorithm 1 seems more confusing, than clarifying:  a) Shouldn't the gradient step be taken in the direction of the gradient of the loss with respect to Theta?  b) There is no description of the variables, most importantly X and f. It is better for the reader to define them in the algorithm than later in the text.  Otherwise, the algorithm definition seems unnecessary.  - Equation (1) is very unclear:   a) Is the purpose to define a loss function or the optimization problem? It seems that it is mixing both.   b) The optimization variable x is defined to be in R^n. Probably it is meant to be in R^k?   c) The constraints notation (s.t. C(A, x)) is rather unusual.    - It is briefly mentioned that an alternating direction method is used to solve the min-min problem. Which method?  - The constraints in equation (2) are identical to the ones in equation (3). They can be mentioned as such to gain space.  - In section 4.1, line 194, K = 10, presumably refers to the number of atoms in the dictionary, namely it should be a small k?  The same holds for section 4.4, line 285.  - In section 4.1, why is the regularizer coefficient gamma set to zero?  Intuitively, structured sparcity should be particularly helpful in finding keypoint correspondences.  What is the effect on the solution when gamma is larger than zero?   The experimental section of the paper seems well written (with a few exceptions, see above). Nevertheless, the experiments in 4.2 and 4.3 each compare to only one existing work.  In general, I can see the idea of the paper has merit, but the carelessness in the formulations in the main part and the lack of comparisons to other works make me hesitant to accept it as is at NIPS.


Regularizing by the Variance of the Activations' Sample-Variances
Normalization techniques play an important role in supporting efficient and often more effective training of deep neural networks. While conventional methods explicitly normalize the activations, we suggest to add a loss term instead. This new loss term encourages the variance of the activations to be stable and not vary from one random mini-batch to the next. As we prove, this encourages the activations to be distributed around a few distinct modes. We also show that if the inputs are from a mixture of two Gaussians, the new loss would either join the two together, or separate between them optimally in the LDA sense, depending on the prior probabilities. Finally, we are able to link the new regularization term to the batchnorm method, which provides it with a regularization perspective. Our experiments demonstrate an improvement in accuracy over the batchnorm technique for both CNNs and fully connected networks.

# Post rebuttal  I believe it is a good submission (7), with some of the issues raised alraedy addressed by the authors rebuttal.   # Original review  This paper describes a new local regulariser for neural networks training to, in principle, reduce the variance of the variance of the activations.   Pros: - an easy to follow line of thought - reasonable mathematical justification of the proposed method - empirical evaluation showing results comparable with batch norm, but without all the problems BN introduces (tracking statistics, heavily affected by batch size, unclear transfer to test time etc.) - A new view of BN as a Kurtosis minimiser (while it could be obvious for some mathematicians, it is definitely a nicely described relation) - Proposed method introduces interesting multi-modality effect to neuron activations  Cons: - Evaluation is minimalistic, while batch norm is indeed extremely popular, it is not the state-of-the-art normaliser, methods like layer normalisation etc. has been shown to outperform it too, consequently it is not clear if proposed method would compare favourably or not. In reviewer's opinion, it is worth considering this paper even if the answer would be negative, as it is an alternative approach, focus on loss, rather than forward pass modification, and as such is much more generic, and "clean" in a sense.  - Change from (6) to (11) is not well justified in the text  Minor comments: - Please use \left and \right operators in equations so that brackets are of correct size (for example in (12) or (13)) - Figures require: a) bigger labels / ticks b) more precise captions, in Figure 1 what do columns represent? - please change "<

Reviewer 3
This paper introduces Variance Constancy Loss (VCL), a novel normalization technique for DNNs. VCL incorporates regularizer loss which penalizes variance of activation’s variance of a DNN layer. VCL allows achieving results which are better or comparable to those of Batch Normalization. These results look very interesting given the fact that VCL adds only a regularizing term and does not modify model workflow as BN does. Another benefit of VCL is that compared to BN it does not require large batches for consistency.  The experimental evaluation covers variety of UCI tasks for fully-connected networks and CIFAR-10/CIFAR-100 data for deep CNN architecture. The VCL is shown to perform better or comparable to BN in terms of final test error. However, it is not clear whether VCL speeds up the training which is a desirable feature of a normalization technique.  The paper is clearly written and easily readable. Although, it seems that there is a typo on line 146 in the inequality “n < 2N”. Instead, there should be “N < 2n” as in the following text it is said that VCL extracts two consecutive subsets of size n from a mini-batch of size N.  Overall, the paper is good. It proposes a novel normalization technique with useful properties which are theoretically grounded and are shown to be effective in practice.   Edits after Author Response:  I keep my rating. The authors have addressed my question in the rebuttal.


The emergence of multiple retinal cell types through efficient coding of natural movies
One of the most striking aspects of early visual processing in the retina is the immediate parcellation of visual information into multiple parallel pathways, formed by different retinal ganglion cell types each tiling the entire visual field. Existing theories of efficient coding have been unable to account for the functional advantages of such cell-type diversity in encoding natural scenes. Here we go beyond previous theories to analyze how a simple linear retinal encoding model with different convolutional cell types efficiently encodes naturalistic spatiotemporal movies given a fixed firing rate budget. We find that optimizing the receptive fields and cell densities of two cell types makes them match the properties of the two main cell types in the primate retina, midget and parasol cells, in terms of spatial and temporal sensitivity, cell spacing, and their relative ratio. Moreover, our theory gives a precise account of how the ratio of midget to parasol cells decreases with retinal eccentricity. Also, we train a nonlinear encoding model with a rectifying nonlinearity to efficiently encode naturalistic movies, and again find emergent receptive fields resembling those of midget and parasol cells that are now further subdivided into ON and OFF types. Thus our work provides a theoretical justification, based on the efficient coding of natural movies, for the existence of the four most dominant cell types in the primate retina that together comprise 70% of all ganglion cells.

The emergence of multiple retinal cell types through efficient coding of natural movies  This papers proposes to study possible distributions of retinal cell types. It builds upon existing literature on four cell types and evaluate how two different configurations of cells could optimally encode natural movies with a unique cell firing rate budget. It draws analogies with optimal configurations and configuration existing in primates. Models are mention to capture 90% of all ganglion cells.  The paper tackle an interesting question on wether retinal cell configurations are mathematically optimized for spatio-temporal stimuli. As a disclaimer, I am not an expert on retinal cells. The paper refers to important literature on the matter and focus on mathematical models of cell distributions. However, little information is provided on the type of target visual stimuli. Naively, one may think that different stimuli should be captured differently across varying cell distributions, and one may question why choosing to study one configuration of cells that optimized with one particular stimuli.   On the model itself, the firing-rate budget is clear. However, the link with neural networks is less clear. Is the neural net used here historically to truly mimic a biological neural network? Could other models be used, is there a particular architecture, number of layers, to follow in such case?   The paper evaluates a the performance of a mixture of 2 types of low/high spatial/temporal cells, and compares it with a configuration of one unique cell types. Why two types and not more configurations?  Minor, what is motivation of scaling (eq 3) - notation of footnote (2) can be confusing (squared?) - footnote (4): empirically? - ref [6]? - there are 8 appendices, pushing the length up for this submission  The authors' feedback proposed clarifications on major points, including on distributions of stimuli (statistics) vs specific stimuli - overall indicating importance of this work on receptive field configurations.


CapProNet: Deep Feature Learning via Orthogonal Projections onto Capsule Subspaces
In this paper, we formalize the idea behind capsule nets of using a capsule vector rather than a neuron activation to predict the label of samples. To this end, we propose to learn a group of capsule subspaces onto which an input feature vector is projected. Then the lengths of resultant capsules are used to score the probability of belonging to different classes. We train such a Capsule Projection Network (CapProNet) by learning an orthogonal projection matrix for each capsule subspace, and show that each capsule subspace is updated until it contains input feature vectors corresponding to the associated class. We will also show that the capsule projection can be viewed as normalizing the multiple columns of the weight matrix simultaneously to form an orthogonal basis, which makes it more effective in incorporating novel components of input features to update capsule representations. In other words, the capsule projection can be viewed as a multi-dimensional weight normalization in capsule subspaces, where the conventional weight normalization is simply a special case of the capsule projection onto 1D lines. Only a small negligible computing overhead is incurred to train the network in low-dimensional capsule subspaces or through an alternative hyper-power iteration to estimate the normalization matrix. Experiment results on image datasets show the presented model can greatly improve the performance of the state-of-the-art ResNet backbones by 10 − 20% and that of the Densenet by 5 − 7% respectively at the same level of computing and memory expenses. The CapProNet establishes the competitive state-of-the-art performance for the family of capsule nets by significantly reducing test errors on the benchmark datasets.

This paper introduces an alternative to CNN based architectures being inspired by the recently proposed capsule networks. The authors proposed to replace the last layer of ResNet variants by a capsule projection network, thereby getting promising results on the CIFAR and SVHN datasets. However, the motivation for using a capsule projection layer is unclear even though the technique is straightforward and easy to implement with minor computational overhead.  The main idea of the capsule projection layer is to project the input feature vector to some learnt capsule subspaces (one for each class in classification setting), which are then used to distinguish between the different classes in classification. The authors also show that this projection technique leads to computation of gradients which are orthogonal to the learnt subspace, enabling discovery of novel characteristics leading to improvement of the learnt subspace. They have also shown interesting visualizations indicating separability of the samples for every classes. The quantitative results in this paper are encouraging when compared with the baselines used.  Strengths : 1. Straightforward idea which is easy to implement with minimal computational overhead. 2. Promising experimental results with interesting visualizations.  Weaknesses: 1. The motivation or the need for this technique is unclear. It would have been great to have some intuition why replacing last layer of ResNets by capsule projection layer is necessary and why should it work. 2. The paper is not very well-written, possibly hurriedly written, so not easy to read. A lot is left desired in presentation and formatting, especially in figures/tables. 3. Even though the technique is novel, the contributions of this paper is not very significant. Also, there is not much attempt in contrasting this technique with traditional classification or manifold learning literature. 4. There are a lot of missing entries in the experimental results table and it is not clear why.  Questions for authors: Why is the input feature vector from backbone network needed to be decomposed into the capsule subspace component and also its component perpendicular to the subspace? What shortcomings in the current techniques lead to such a design? What purpose is the component perpendicular to the subspace serving? The authors state that this component appears in the gradient and helps in detecting novel characteristics. However, the gradient (Eq 3) does not only contain the perpendicular component but also another term x^T W_l^{+T} - is not this transformation similar to P_l x (the projection to the subspace). How to interpret this term in the gradient?  Moreover, should we interpret the projection onto subspace as a dimensionality reduction technique? If so, how does it compare with standard dimensionality reduction techniques or a simple dimension-reducing matrix transformation? What does "grouping neurons to form capsules" mean - any reference or explanation would be useful? Any insights into why orthogonal projection is needed will be helpful. Are there any reason why subspace dimension c was chosen to be in smaller ranges apart from computational aspect/independence assumption? Is it possible that a larger c can lead to better separability?  Regarding experiments, it will be good to have baselines like densenet, capsule networks (Dynamic routing between capsules, Sabour et al NIPS 2017 - they have also tried out on CIFAR10). Moreover it will be interesting to see if the capsule projection layer is working well only if the backbone network is a ResNet type network or does it help even when backbone is InceptionNet or VGGNet/AlexNet.


Kalman Normalization: Normalizing Internal Representations Across Network Layers
As an indispensable component, Batch Normalization (BN) has successfully improved the training of deep neural networks (DNNs) with mini-batches, by normalizing the distribution of the internal representation for each hidden layer. However, the effectiveness of BN would diminish with the scenario of micro-batch (e.g. less than 4 samples in a mini-batch), since the estimated statistics in a mini-batch are not reliable with insufficient samples. This limits BN's room in training larger models on segmentation, detection, and video-related problems, which require small batches constrained by memory consumption. In this paper, we present a novel normalization method, called Kalman Normalization (KN), for improving and accelerating the training of DNNs, particularly under the context of microbatches. Specifically, unlike the existing solutions treating each hidden layer as an isolated system, KN treats all the layers in a network as a whole system, and estimates the statistics of a certain layer by considering the distributions of all its preceding layers, mimicking the merits of Kalman Filtering. On ResNet50 trained in ImageNet, KN has 3.4% lower error than its BN counterpart when using a batch size of 4; Even when using typical batch sizes, KN still maintains an advantage over BN while other BN variants suffer a performance degradation. Moreover, KN can be naturally generalized to many existing normalization variants to obtain gains, e.g.equipping Group Normalization [34] with Group Kalman Normalization (GKN). KN can outperform BN and its variants for large scale object detection and segmentation task in COCO 2017.

This paper addresses the batch normalization problem with small batch size. Traditional batch normalization relies on the estimation of the batch statistics (mean and variance) over each batch. Batch normalization requires relatively large batch size in order to obtain relatively reliable estimates of these statistics. However, due to memory constraint, some higher-level tasks (e.g., in computer vision) could not use large batchsize. Therefore, developing normalization method with small batch size is important for improving the performance of these systems. Experient results show that the proposed KN outperform previous batch normalization approaches both in regular batch sizes and small batch sizes.  The problem formulation and the proposed algorithm have a gap from the canonical Kalman filter and it is not discussed clearly in the paper. It is a bit confusing about the formulation of the Kalman filter. What is the state x^k in (2)? Is it the activation vector at the k-th hidden layer or the groundtruth mean at the hidden layer? I think a correct formulation of Kalman filter for this problem is to model the state as the true mean value if the observation is modeled to be the (noisy) observation of this mean and u^k is the disturbance due to inaccurate modeling with a linear dynamic system. Otherwise, if the state is modeled to be the realization of the hidden activation vector at the k-th layer, then \hat{\mu}_{k|k}, which in Kalman filter context is the conditional mean of the state given all the observations up to k, is an minimum-mean-square-error (MMSE) estimate of the realization of the activation vector. In batch normalization problem, the desired quantity to estimate should be the mean not the realization of the activation vector. So the current formulation in (2) is not correct. Furthermore, the proposed algorithm is quite different from standard Kalman filter, where the gain matrix (instead of a gain scalar q^k) is computed from a Riccati equation (a nonlinear recursion over the covariance matrix). None of these are sufficiently discussed.


Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behavior
Inferring intent from observed behavior has been studied extensively within the frameworks of Bayesian inverse planning and inverse reinforcement learning. These methods infer a goal or reward function that best explains the actions of the observed agent, typically a human demonstrator. Another agent can use this inferred intent to predict, imitate, or assist the human user. However, a central assumption in inverse reinforcement learning is that the demonstrator is close to optimal. While models of suboptimal behavior exist, they typically assume that suboptimal actions are the result of some type of random noise or a known cognitive bias, like temporal inconsistency. In this paper, we take an alternative approach, and model suboptimal behavior as the result of internal model misspecification: the reason that user actions might deviate from near-optimal actions is that the user has an incorrect set of beliefs about the rules -the dynamics -governing how actions affect the environment. Our insight is that while demonstrated actions may be suboptimal in the real world, they may actually be near-optimal with respect to the user's internal model of the dynamics. By estimating these internal beliefs from observed behavior, we arrive at a new method for inferring intent. We demonstrate in simulation and in a user study with 12 participants that this approach enables us to more accurately model human intent, and can be used in a variety of applications, including offering assistance in a shared autonomy framework and inferring human preferences.

The paper investigates the problem of inferring an agent's belief of the system dynamics of an MDP, given demonstrations of its behavior and the reward function it was optimizing. Knowledge of this internal belief can be used for Inverse Reinforcement Learning of an unknown task in the same environment. Furthermore, given the action provided by the agent, its intended action on the true dynamics can be inferred. This allows for assistive tele-operation, by applying the intended actions to the system instead of the provided ones. The proposed method models the agent using the model derived in maximum causal entropy inverse reinforcement learning. The internal belief of the system dynamics is inferred by maximizing the likelihood of the demonstrations with respect to the parameters of the dynamic model. Human studies on the lunar lander Atari game demonstrate that such assistive systems based on inferred internal models can help the user to solve the given task.  Related problems in an MDP are RL, which learns behavior from dynamics and reward, as well as IRL, which learns reward from dynamics and behavior. This work tackles the problem of learning dynamics from behavior and reward which has gotten little attention from the community. There has been some prior work on this problem that was also mentioned in the paper. Herman et al. (2016) considered the problem of simultaneously inferring dynamics and reward function from expert demonstrations while also building on MaxCausalEnt-IRL. Although, the work by Herman et al. allows for internal models that differ from the true dynamics, they focused on inferring the true dynamics and assumed in the experiments that the internal model was correct. Golub et al. (2013) infer an internal model from reward function and dynamics. They derived an EM-algorithm that they applied for learning the internal model of Rhesus monkeys that were using a brain computer interface. However they consider a slightly different scenario where the subject's internal estimate of the state differs from the true state due to sensory feedback delay. The paper under review focuses on the internal model (and assumes that the true dynamics are known for the action intent prior) and in contrast to previous work puts more focus on the use cases. Using an internal model to correct the user's actions is novel to my knowledge and could be useful for real applications like tele-operating a robot. Also the application to IRL is interesting, however I think this scenario is less relevant, because we assume that 1) we want to infer the reward function of an agent that 2) is willing to perform demonstrations on different tasks and we 3) assume that its internal model is significantly different from the true dynamics.   The proposed method is based on a sound optimization problem by maximizing the likelihood of the demonstrations assuming the expert model of MaxCausalEnt-IRL. Yet, I think it would be more principled to derive both the expert model as well as the objective from a single optimization problem (e.g. similar to MaxEnt Reinforcement Learning but optimizing dynamics rather than policy).  This might also avoid the need for the penalty method, which is a simple but rather inefficient and unstable approach. The proposed method does in general tend to prefer the easy route, e.g. by assuming deterministic dynamics (although humans seem to take uncertainty into account [1]) and by using uniform sampling in section 3.1. which should lead to rather high variance. Nevertheless, the method is still sound and, in my opinion, the method itself is not the main contribution of the paper anyway. Indeed, it is more important to raise the awareness and show the usefulness of such little explored research branches, which is I think the main contribution of the proposed paper.  The paper is well written and easy to follow. However, I think it would be better if the experiments would be better separated, e.g. by removing everything relating to the lunar lander game from section 5.1 and calling the section something like "Gridworld experiments". The gridworld experiments should be briefly described also in the paper itself.  Questions to the authors: For learning the internal model, we need the user to optimize a given reward function. For some problems, specifying a reward is easy (e.g. Atari games). However, how would you make a human optimize a reward according to maxent-model for a real application?  Concerning the action intend prior: Wouldn't it be simpler to 1) solve the MDP using the true dynamics and 2) compute the mapping (f_theta) from actions to intended actions as the maximum likelihood conditional distribution?  How are the demonstrations on the gridworld collected? If whole trajectories were collected at once, I assume that the user knew how the dynamics are scrambled since the proposed method can not handle changing beliefs. However, if the user knew the true dynamics, why would his internal belief differ? Wouldn't the sub-optimality of the user's behavior be caused solely by its inability to choose the correct actions under real-time constraints? Or were the trajectories collected by independently querying the users action for a given state without revealing the consequences of this action?  [1] Belousov, Boris, et al. "Catching heuristics are optimal control policies." Advances in Neural Information Processing Systems. 2016.  Edit: I read the the other rebuttal / reviews and did not see a reason to change my assessment.


How Much Restricted Isometry is Needed In Nonconvex Matrix Recovery?
When the linear measurements of an instance of low-rank matrix recovery satisfy a restricted isometry property (RIP)-i.e. they are approximately normpreserving-the problem is known to contain no spurious local minima, so exact recovery is guaranteed. In this paper, we show that moderate RIP is not enough to eliminate spurious local minima, so existing results can only hold for near-perfect RIP. In fact, counterexamples are ubiquitous: we prove that every x is the spurious local minimum of a rank-1 instance of matrix recovery that satisfies RIP. One specific counterexample has RIP constant δ = 1/2, but causes randomly initialized stochastic gradient descent (SGD) to fail 12% of the time. SGD is frequently able to avoid and escape spurious local minima, but this empirical result shows that it can occasionally be defeated by their existence. Hence, while exact recovery guarantees will likely require a proof of no spurious local minima, arguments based solely on norm preservation will only be applicable to a narrow set of nearly-isotropic instances.

Summary It has recently been proved that simple non-convex algorithms can recover a low-rank matrix from linear measurements, provided that the measurement operator satisfies a Restricted Isometry Property, with a good enough constant. Indeed, in this case, the natural objective function associated with the problem has no second-order critical point other than the solution. In this article, the authors ask how good the constant must be so that this property holds. They propose a convex program that finds measurement operators satisfying RIP, but for which the objective function has "bad" second-order critical points. They analyze this program in the case where the matrix to be recovered has rank 1, and deduce from this analysis that, for any x,z, there exists a RIP operator such that x is a bad second-order critical point when the true unknown is zz*; they upper bound the RIP constant. Finally, they run their program, and study through a few examples the behavior of Stochastic Gradient Descent on instances generated by it.   Main remarks The main things I appreciated in this article are: 1. I found the topic really interesting. There has been a lot of works, in the recent years, on why simple non-convex heuristics are able to efficiently solve some a priori difficult problems. Understanding which properties the problems must satisfy so that this is possible is an almost completely open problem. The present article studies, and relatively precisely describes, an aspect of this question. 2. I appreciated the approach used to tackle the problem, with this convex program that allows for both a theoretical and a numerical analysis. It was new to me, and proved efficient to answer the quesiton solved by the authors. 3. The writing style is clear and easy to read.  However, there are also, in my opinion, some issues: 1. I think the proof of Theorem 7 should absolutely be corrected and rewritten. I do not say that it is not true, but there are many things in it that I do not understand: - I do not see why naively repeating the prof of Theorem 5 leads to the equation after line 422, because I do not see why the positive eigenvalues of L^T(y)-M^T(U3) should be essentially the same as in Lemma 8 (the presence of X U3 X^T bothers me). Additionally, u3 should be defined. - Lines 431-432: it is g(0)=sin theta and g'(0)=cos theta, so I do not get the equation that follows. - In Lemma 10, it seems to me that g(beta) should be defined as a max, not a min (minimizing the objective function is equivalent to maximizing cos theta). - In the equation after line 457, why is it "+ V2 beta" and not "- V2 beta"? And I do not understand the second equality. Do we have alpha^2 + beta^2 = 1? - Line 458: Why is it sufficient to show the orthogonality? - Line after 459: A line of explanation as to why the optimal y and U solve this problem would be welcome, and the \tilde u_i should be defined. 2. Some aspects of the presentation could be improved. In particular, - Any measurement operator that is injective on the set of rank-r matrices satisfies 2r-RIP for some constant stricly smaller than one. But given that it is in general NP-hard to recover a low-rank matrix from linear measurements, it is somewhat expected that, for some RIP matrices, spurious second-order critical points (or at least points that look very much like spurious second-order critical points) exist. Consequently, it seems to me that it is not a surprise that the answer to the question on lines 58-59 is negative. Subsection 1.1 does not make this clear. - It is a naive remark, but 0 is always a critical point of the objective function; the discussion after Theorem 5 seems to ignore this. - Subsection 1.3: There are proofs of correctness for non-convex methods that do not really rely on properties very close to RIP. This is notably the case in phase retrieval, where the RIP does not hold. 3. It seems to me that some natural questions raised by this work could have been explored. - Experiment 2 shows that SGD avoids local minima more or less often, depending on their configuration. It would have been interesting to suggest explanations as to the factors that control this phenomenon. If I understood correctly, an obvious difference between the first and second examples is that, in the first case, the spurious local minimum has norm 2, while in the second one, it has norm 1/sqrt(2). The initialization strategy gives more weight to vectors with small norm. Does this explain part of the difference between the two examples? Can the autors propose and test other hypotheses? - About the tightness of the bound in Experiment 1, it seems to me that it should be possible to get a lower bound on the optimal delta, by replacing, in Equation (10), the equation "eta I <= H <= I" by "eta I <= P^T H P <= I", where the columns of P form an orthonormal basis of a subspace of S^n whose matrices all have rank at most 2r (for example, the matrices in S^n whose range is included in the vector space generated by the ranges of x and z). Is it correct? Have the authors tested it?   Minor remarks - Theorem 2: "local minimum" -> "second-order critical point" - Do the authors have ideas on how to generate operators A that satisfy the same properties, but for which m < n(n+1)/2? - Subsection 1.3: It is surprising to say that convex approaches enjoy stronger guarantees, then that some can be shown to always succeed when delta_(4r) < 0.4142 or delta_(4r) < 0.1907 (conditions which are not really weaker that requiring delta_(2r) < 0.2). - Line 132: "require" -> "requires" - The formulation in line 156 slightly suggests that the condition number is a convex function of H (which is not the case). - There are typos in Equation (6) (an "A" and a factor 2 missing, I think). - Equation (10): isn't the condition H \succeq 0 redundant? - In Section 3 and in the associated proofs, it would be better to remove all occurences of the variable "r", since only the r=1 case is studied. - Line 222: It seems to me that Example 3 only shows that the smallest delta achievable is smaller than or equal to 1/2, not that it is exactly 1/2. - Right sides of Figures 2 and 3: what does the y axis represent? - Line 243: "the the" -> "the". - Bibliography: "Candès" sometimes has an accent, sometimes not. - Line 387: There is a word missing after "coincide". - Lines 389-392: I do not think it is true that U1 takes on the positive eigenvalues, and U2 the negative ones. Imagine that Lambda = Diag(0.5,-0.5). Then U1 = V Diag(1,0) V^T and U2 = V Diag(0.5,0.5) V^T satisfy the constraints, and minimize the trace of U2. - Line 395: "eigenavlues" -> "eigenvalues" - Equation after Line 398: there is a theta missing in the denominator. - Line 436: the expression for M^T is not correct. - Line 442: there are typos in the expression for the eigenvalues. - Theorem 5 and Lemma 9: What is the interest of introducing the lower bound sin^2 phi / sqrt(1+rho^4) for sin theta? The exact expression that comes out of Lemma 9 is not so much more complex. - Equation after Line 450: There is a transpose missing. - Equation after Line 464: There are typos. - Equations in the proof of Lemma 11: v should be defined.   Added after author feedback: Many thanks to the authors for their clear and detailed response. Almost all of my remarks seem to have been taken into account. I changed my score accordingly.


Unsupervised Image-to-Image Translation Using Domain-Specific Variational Information Bound
Unsupervised image-to-image translation is a class of computer vision problems which aims at modeling conditional distribution of images in the target domain, given a set of unpaired images in the source and target domains. An image in the source domain might have multiple representations in the target domain. Therefore, ambiguity in modeling of the conditional distribution arises, specially when the images in the source and target domains come from different modalities. Current approaches mostly rely on simplifying assumptions to map both domains into a shared-latent space. Consequently, they are only able to model the domain-invariant information between the two modalities. These approaches usually fail to model domain-specific information which has no representation in the target domain. In this work, we propose an unsupervised image-to-image translation framework which maximizes a domain-specific variational information bound and learns the target domain-invariant representation of the two domain. The proposed framework makes it possible to map a single source image into multiple images in the target domain, utilizing several target domain-specific codes sampled randomly from the prior distribution, or extracted from reference images.

Summary:  The main contribution of this paper is a new framework for unsupervised image-to-image translation. Specifically, instead of learning only the domain-invariant information between two domains, the proposed approach extends CycleGAN to learn a domain-specific code for each modality and a domain-invariant code shared by the two modalities. The domain-specific code is learned via a domain-specific variational information bound. The trained model can be used for unsupervised one-to-many image-to-image translation.  Strengths:  (1) Significance: Though unsupervised image-to-image translation has been studied before, this paper pointed out one common drawback of such methods, i.e., no domain-specific code is learned for each individual modality. I think this is a good point, and the proposed method nicely solves this problem, and thus has some significance. The topic is also suitable for general NIPS readers.   (2) Originality: The proposed model is interesting and seems novel. I like the idea of decomposing the latent code into the domain-invariant and domain-specific parts. The idea of using a domain-specific variational information bound to enforce the model disentangle the latent code also seems interesting.   Weaknesses:  (1) Clarity: I like the first three sections, which introduces the problem and proposed approaches clearly. However, Section 4 is generally not well written, and not easy to follow. Please see the Questions section for more details.   For presentation, I think the introduction part talks about a little bit too many details. For example, it is not clear to me why putting Figure 1 & 2 in Section 1, and it seems to me moving these two figures to the experimental section and discussing the figures there might be better.  Also, in Table 1, the authors showed the human evaluation results. However, how this human evaluation is performed is not discussed. I would recommend describing this in the main text.   (2) Quality: I think the idea is novel, but the authors did not introduce the proposed variational information bound in Section 4 clearly, which seems to be a key contribution of this paper. Also, the authors did not provide enough experimental details for other people to reproduce the experimental results. Specifically, I have the following questions.      Questions:  (1) It is not clear to me why changing from Eqn. (2) to Eqn. (7), i.e., replacing KL with MMD will maximize the mutual information between x and v_{x1}. Please explain it clearly. I do not get the argument stated in Lines 166 to 169.  (2) I understand that the bound in Eqn. (8) is to make the mutual information trackable. However, the formulation of the variational approximation r(\hat{v}_{y1}) is never introduced. Please clarify this.    (3) There are just too many hyper-parameters in the final proposed method. For example, \lambda_{1,2,3}, \beta, \alpha_{1,2,3,4}^{1,2}, the bandwidth value \sigma in the Gaussian kernel of MMD. I'd like to know how these hyperparameters are selected, how sensitive these numbers are, and what are the numbers being used in the experiments. For the current draft, I think it will be difficult to reproduce the experiments.   Overall, I think this paper is marginally below the acceptance threshold. Though the idea is novel, the presentation is not satisfying. If Section 4 is described more clearly, it has a chance to be a good paper.   Minor issues:  (1) Line 13: "domain" => "domains", Line 77: "Works" => "Work"  (2) Equation (2) & (7): the second expectation term: "v_{x1} \sim p(v_{x1}|x)" => "v_{x1} \sim q_x(v_{x1}|x)"   (3) It seems to me Figure 5 & 6 demonstrates the same thing. I am not sure why plotting these two figures separately here. I suggest combining these two figures, and getting rid of one of Figure 5(b) and Figure 6(c). These two subfigures basically are the same.   -- After Rebuttal --  I have read the rebuttal and other reviewers' comments. Generally, I think the authors partially solved my concerns. For example, they discussed how they select hyper-parameters, and more technical details of the method are provided. As to quantitative analysis, it seems using LPIPS and fooling rate aligns well with the previous work. I feel this paper has some novelty inside, and in the camera-ready version, as the authors claim, I hope the authors will take the comments into consideration to make the presentation more clear.  I slightly prefer to voting acceptance, and therefore moving the score to a 6.


Deep Poisson gamma dynamical systems
We develop deep Poisson-gamma dynamical systems (DPGDS) to model sequentially observed multivariate count data, improving previously proposed models by not only mining deep hierarchical latent structure from the data, but also capturing both first-order and long-range temporal dependencies. Using sophisticated but simple-to-implement data augmentation techniques, we derived closed-form Gibbs sampling update equations by first backward and upward propagating auxiliary latent counts, and then forward and downward sampling latent variables. Moreover, we develop stochastic gradient MCMC inference that is scalable to very long multivariate count time series. Experiments on both synthetic and a variety of real-world data demonstrate that the proposed model not only has excellent predictive performance, but also provides highly interpretable multilayer latent structure to represent hierarchical and temporal information propagation.

This paper presents Deep Poisson-Gamma Dynamical System (DPGDS) for modeling temporal multivariate count data. It is based on previously developed Gamma-Belief networks, extended to the dynamical scenarios by adding transitions of latent units in consecutive times. The paper is well written, and connections to previous papers are explained clearly.  Major comments: 1. While the temporal structure is based on transition of latent units in a Markov manner, the authors' claim about better capturing long-range temporal changes should be justified more clearly.  2. As the shape parameter in the Gamma distributions corresponding to latent units is summation of two terms, does imposing Dirichlet distributions on \Phi and \Pi makes the latent units completely identifiable?  3. I suggest to compare your performance with dynamical methods based on Gaussian Process Latent Variable Models, which are incorporated specifically to have more flexible temporal structure, if possible.  Minor comments: 1. In line 30, "separated" should be replaced by "separately". 2. The sentence starting at line 46 is better to be rewritten more clearly.


Manifold-tiling Localized Receptive Fields are Optimal in Similarity-preserving Neural Networks
Many neurons in the brain, such as place cells in the rodent hippocampus, have localized receptive fields, i.e., they respond to a small neighborhood of stimulus space. What is the functional significance of such representations and how can they arise? Here, we propose that localized receptive fields emerge in similarity-preserving networks of rectifying neurons that learn low-dimensional manifolds populated by sensory inputs. Numerical simulations of such networks on standard datasets yield manifold-tiling localized receptive fields. More generally, we show analytically that, for data lying on symmetric manifolds, optimal solutions of objectives, from which similarity-preserving networks are derived, have localized receptive fields. Therefore, nonnegative similarity-preserving mapping (NSM) implemented by neural networks can model representations of continuous manifolds in the brain.

In this work, the authors show that similarity-preserving, the problem of creating an encoding of inputs on a manifold such that the similarity between encoded outputs between inputs matches similarity between inputs, yields localized receptive field that tile the space. They present a biologically plausible online neural learning model which combines several neural elements in a way that is equivalent to optimizing performance on this problem, and thus learns localized receptive fields. Additionally, they apply these methods to generate localized receptive fields over real-world datasets, such as MNIST and images of rotated objects.   Overall I found this to be a good paper and enjoyed reading it.   Quality:  The paper starts from the basic, reasonable idea of similarity-preserving transformations, and then proceeds to prove mathematically, using sophisticated techniques, that the optimal receptive fields are localized, as well as show that these can be optimized by a biologically plausible learning rule.   I view this as a significant result, although I don’t know the literature so well.  The experimental results with MNIST and rotated objects are interesting, especially in that the manifold topology of inputs matches the manifold topology of encodings, but there is a somewhat preliminary feel to them, in that there is not much information about how well this scheme is able to generalize to unknown examples.   Clarity:  While I was not able to follow all proofs and mathematical detail, the authors use some fairly heavy-duty techniques, so I don’t think that’s a fair criticism. Given how dense the proof techniques are, I feel that the paper would be significantly improved if the authors took a more pedagogical/intuition-providing approach in general.  For example, in Line 130, I have to admit I did not understand the proof of theorem 1, although it seems quite technical. I have some intuition for why this is true, but it might be helpful for the authors to provide some intuition. For example, there might be a much simpler an intuitive proof for the ring in the continuum limit; would it be possible for the authors to demonstrate this special case in a more pedagogical manner? line 184 Section 5: After staring for a little while, I saw how NSM-1 yields equation 11, but still don’t understand how the authors get from equation (11) to equation (12). The authors cite:  Cengiz Pehlevan, Anirvan M Sengupta, and Dmitri B Chklovskii. Why do similarity matching objectives lead to hebbian/anti-hebbian networks? Neural computation, 30(1):84–124, 2018  nearby but there seems is no proof there either. Can the authors add a section in the supplemental proving that this is true and add more intuition behind what each of these terms are? Some intuition is provided in the main, but it is quite cursory given space limitations.  During online optimization(line 188), what values were used for relative learning rates gamma and eta? My understanding of equations (13) (14), is that each step relies on a *single* sample x_t. Is this correct? If so, is the manner in which the manifold is explored important? For example, does it depend if x_{t}, x_{t+1} tend to be similar or unrelated?  There seem to be other details missing, which is why I gave the current manuscript a medium reproducibility score rather than a high one. I suspect these details can be fixed and filled in without too much trouble.  Originality and Significance:  While I don’t claim to have a deep knowledge of the literature, this work seems to be original and significant. Receptive fields localized in physical/configuration space are found in many places, and understanding their function as well as their formation through the lens of similarity-preservation might be an important insight.   There seem to be analytical results that exist concerning related similarity-preserving optimization problems, but not this particular one involving a cutoff where nearly-orthogonal inputs should yield completely-orthogonal encodings.    Miscellaneous:  In figure 4, how is the receptive field width and density (Y transpose, third matrix) determined by the input correlation structure (X transpose X, first matrix)? Can the authors provide some intuition behind this? It would also be nice if the authors provided some intuition/supplementary figures for how receptive fields depend on various parameters in the manner of SI figure 6.


Data-Driven Clustering via Parameterized Lloyd's Families
Algorithms for clustering points in metric spaces is a long-studied area of research. Clustering has seen a multitude of work both theoretically, in understanding the approximation guarantees possible for many objective functions such as k-median and k-means clustering, and experimentally, in finding the fastest algorithms and seeding procedures for Lloyd's algorithm. The performance of a given clustering algorithm depends on the specific application at hand, and this may not be known up front. For example, a "typical instance" may vary depending on the application, and different clustering heuristics perform differently depending on the instance. In this paper, we define an infinite family of algorithms generalizing Lloyd's algorithm, with one parameter controlling the initialization procedure, and another parameter controlling the local search procedure. This family of algorithms includes the celebrated k-means++ algorithm, as well as the classic farthest-first traversal algorithm. We design efficient learning algorithms which receive samples from an application-specific distribution over clustering instances and learn a nearoptimal clustering algorithm from the class. We show the best parameters vary significantly across datasets such as MNIST, CIFAR, and mixtures of Gaussians. Our learned algorithms never perform worse than k-means++, and on some datasets we see significant improvements.

The paper proposes a generalization of the KMeans++ Algorithm by introducing non-negative parameters alpha and beta. alpha parameterizes the initialization phase, i.e. the distribution from which the initial center seeds are sampled. beta parameterizes the objective according to which the data is clustered.  The motivation is that different instances of clustering problems may cluster well according to different clustering objectives. The optimal parameter configuration of alpha and beta defines an optimal choice, from the proposed family of clustering algorithms.  The paper offers several theoretical contributions. It provides guarantees for the number of samples necessary such that the empirically best parameter set yields clustering costs is within epsilon bounds of the optimal parameters.  It provides an algorithm for the enumeration of all possible sets of initial centers for any alpha-interval. This allows to provide runtime guarantees for tuning the parameter alpha.  The proposed algorithm is evaluated on MNIST, Cifar10, CNAE-9 and a mixture of Gaussians dataset, where the datasets are split into training and test sets. Results show that the optimal parameters transfer from one training to test set.  The theoretical contribution of this paper seems solid. Proofs are provided for all theorems.  The fact that a cost close to the empirically optimal value can be achieved for parameter beta = 2 for all datasets (see supplementary material), equaling to the k-means objective, is a bit disappointing. However, this finding might be different for different datasets.  In general, the paper is difficult to read. This might be due to the fact that many details do not find place in the main paper, such that, without reading the proofs in detail, it becomes hard to understand the exact conditions under which the provided guarantees hold. For example  on page 4, line 182, the paper need to be more precise! The assumption made is that ALL data points from one cluster have to be closer to ALL data points of the same cluster than to any point of a different cluster (which is a very strong assumption in practice).


Learning Beam Search Policies via Imitation Learning
Beam search is widely used for approximate decoding in structured prediction problems. Models often use a beam at test time but ignore its existence at train time, and therefore do not explicitly learn how to use the beam. We develop an unifying meta-algorithm for learning beam search policies using imitation learning. In our setting, the beam is part of the model, and not just an artifact of approximate decoding. Our meta-algorithm captures existing learning algorithms and suggests new ones. It also lets us show novel no-regret guarantees for learning beam search policies.

This paper develops a unifying formalism (a meta-algorithm) for imitation learning in the context of beam search; that is, learning in a beam-search-aware manner with access to an oracle that will tell you the minimum completion cost for any partial state. This formalism exposes a subtle but important corner of the space that has not yet been explored: algorithms that are beam aware, but which do not reset or stop after the gold-standard falls out of the beam, but instead continue along the best possible remaining path. Using this formalism, they are able to develop regret guarantees for this new case, alongside novel guarantees for existing cases where the algorithm stops or resets. They show the case where the algorithm continues to have better regret guarantees.  This paper is a masterwork of notation. No symbol is wasted, no symbol is under-explained. Everything is crisp and precise. The paper is demanding, but with some patience, its clarity is remarkable. Its unifying framework builds an effective bridge between the worlds of beam-search aware structured prediction and imitation learning, and for this reviewer, that bridge is very valuable. The only thing I would have liked to see, which was missing from both the main paper and the appendices, is the inclusion of a table matching existing papers to specific loss-strategy combinations in the meta-algorithm.   As implied in my summary above, the unexplored corner of imitation learning that they’ve uncovered with this formalism (that is, beam + continue with oracle costs), is novel and interesting. Even ignoring the regret guarantees, from an intuitive standpoint, it is likely to be useful for practitioners down the road, once we have some experiments supporting this choice (this paper has no experiments, which I assume is okay for NIPS).   Some nit-picky concerns:  The two points laid out in the first paragraph: (1) learning ignores the beam and (2) learning uses only oracle trajectories, are a good way to spell out the standard maximum likelihood training process. However, to say that your method is the only method the address both issues simultaneously is too strong. Any beam-aware method naturally addresses (1) and (2), as by definition it does not use *only* oracle trajectories. You need a better way of phrasing number (2) to emphasize that you are the proposing the first method to both use a beam and oracle costs to avoid stopping or resetting when the gold-standard falls out of the beam - which is a fair amount more subtle than being the first to use both a beam and non-oracle trajectories. This becomes clear two paragraphs later, but it is misleading for the first paragraph of the paper.  The main text mentions Thm 4 without mentioning that it can only be found in the appendix.  --  Dropping in to thank the authors for their response, and for including the table I requested.


MetaAnchor: Learning to Detect Objects with Customized Anchors
We propose a novel and flexible anchor mechanism named MetaAnchor for object detection frameworks. Unlike many previous detectors model anchors via a predefined manner, in MetaAnchor anchor functions could be dynamically generated from the arbitrary customized prior boxes. Taking advantage of weight prediction, MetaAnchor is able to work with most of the anchor-based object detection systems such as RetinaNet. Compared with the predefined anchor scheme, we empirically find that MetaAnchor is more robust to anchor settings and bounding box distributions; in addition, it also shows the potential on transfer tasks. Our experiment on COCO detection task shows that MetaAnchor consistently outperforms the counterparts in various scenarios.

Summary:  This paper proposes MetaAnchor, which is a anchor mechanism for object detection. In MetaAnchor, anchor functions are dynamically generated from anchor box bi, which describes the common properties of object boxes associated with i_th bin. It introduces a anchor function generator which maps any bounding box prior bi to the corresponding anchor function. In this paper, the anchor function generator is modeled as two-layer network for residual term R, added to the shared and learnable parameters for the anchor function theta^*. The residual term R can also depends on input feature x, which introduces the data-dependent variant of anchor function generator. Using weight prediction mechanism, anchor function generator could be implemented and embedded into existing object detection frameworks for joint optimization. Experiments to apply MetaAnchor to state-of-the-art single-stage detector RetinaNet on COCO dataset shows that MetaAnchor outperforms consistently outperforms the counterparts in various scenarios.  Strengths:  - The technical content of the paper appears to be correct with support of some experiment results. - This paper is clearly written with many details description and discussion on how optimization works and how experiment is performed. - There is some novelty of using neural network to learn anchor function.  Weaknesses:  - As mentioned in section 3.2, MetaAnchor does not show significant improvement for two-stage anchor-based object detection. - The experiment evaluation is only done for one method and on one dataset. It is not very convincing that MetaAnchor is able to work with most of the anchor-based object detection system.  Rebuttal Response:  Overall, I think this approach has value on one-stage frameworks. The experiment is only on COCO, but performed in an extensive way. It would be great to have more consistently good results on another benchmark dataset such as OpenImage. But for this submission now, I am willing the upscale my score from 4 to 6.