Generative Adversarial Nets
We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and D equal to 1 2 everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with backpropagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples.

Review of submission 1384:Generative Adversarial NetsSummary: There are two "adversarial" MLPs G and D. G gets random inputs, and should learn to generate samples of a given data distribution. D is trained to learn the probability that something like G's output stems from the original distribution instead of G, while G is trained "to maximize the probability of D making a mistake." First experimental results are described. Comments:Sections 1 and 2:I find the basic ideas quite interesting and appealing.The discussion of related work, however, could be a bit more balanced. Example:"So far, the most striking successes in deep learning have involved discriminative models, usually those that map a high-dimensional, rich sensory input to a class label [14, 21]."References [14, 21] are about discriminative models for speech recognition and Imagenet. But [14] does not yet include the current state of the art of discriminative models for speech - compare instead the work of Graves et al, e.g., IJCAI 2007, ICASSP 2013, ICML 2014. And [21] is not any longer state of the art for Imagenet - compare instead work of Zeiler and Fergus http://arxiv.org/abs/1311.2901  Also, the cited survey of 2009 [1] on "rich, hierarchical models [1]" does not mention numerous original papers on deep learning with hierarchical models since the 1960s, many of them listed in a recent comprehensive overview http://arxiv.org/abs/1404.7828 In the context of "generalized denoising auto-encoders [4]", one should also cite the earlier, closely related work of Behnke at IJCAI 2001 and ICANN 2003.In the context of backpropagation "into a stochastic network with continuous activations (so that gradients can be back-propagated)," one should also cite the much older original papers on this, in particular those by Williams (e.g., Machine Learning, 1992), who adjusted mean and variance of probabilistic continuous units by backpropagation through random number generators.Finally, how is the submission related to the first work on "adversarial" MLPs for modeling data distributions through estimating conditional probabilities, which was called "predictability minimisation" or PM (Schmidhuber, NECO 1992)? The new approach seems similar in many ways. Both approaches use "adversarial" MLPs to estimate certain probabilities and to learn to encode distributions. A difference is that the new system learns to generate a non-trivial distribution in response to statistically independent, random inputs, while good old PM learns to generate statistically independent, random outputs in response to a non-trivial distribution (by extracting mutually independent, factorial features encoding the distribution). Hence the new system essentially inverts the direction of PM - is this the main difference? Should it perhaps be called "inverse PM"?Sections 4 and 6 - convergence:Interesting theorem: Under certain assumptions the algorithm will converge:"If G and D have enough capacity, and at each step of Algorithm 1, D is allowed to approximately track its optimum given G, and each update of G improves its criterion, then Algorithm 1 converges to the global optimum."Here I seem to stumble about a possibly irrelevant or trivial issue. Is it really enough that each update of G improves its criterion? Could it not be that those improvements get exponentially smaller over time in some weird way, such that even infinitely many of them might converge to a suboptimal point? My intuition tells me that this is not the case, and probably this is just a minor technical issue, but has this really been shown, or is it trivial for some reason - if so, is there a reference for this?As was to be expected, some of the drawbacks mentioned by the authors are similar to those of adversarial PM for distribution modeling: "G must not be trained too much without updating D ..."Section 5 - experimentsExperimental results and graphical visualizations are interesting. They seem a bit hard to judge tough, for lack of direct comparisons to the state of the art, although at least for one of the experiments (Parzen density) the authors cautiously write: "This method of estimating the likelihood has somewhat high variance and does not perform well in high dimensional spaces but it is the best method available to our knowledge."The authors comment on the experimental results with some handwaving:"While we make no claim that these samples are better than samples generated by existing methods, we believe that these samples are at least competitive with the better generative models in the literature and highlight the potential of the adversarial framework."Section 7 - a somewhat half-hearted conclusion:"This paper has demonstrated the viability of the adversarial modeling framework, suggesting that these research directions could prove useful."


On Integrated Clustering and Outlier Detection
We model the joint clustering and outlier detection problem using an extension of the facility location formulation. The advantages of combining clustering and outlier selection include: (i) the resulting clusters tend to be compact and semantically coherent (ii) the clusters are more robust against data perturbations and (iii) the outliers are contextualised by the clusters and more interpretable. We provide a practical subgradient-based algorithm for the problem and also study the theoretical properties of algorithm in terms of approximation and convergence. Extensive evaluation on synthetic and real data sets attest to both the quality and scalability of our proposed method.

The paper describes an approach to clustering which deals automatically with the problem of outlier detection. The whole idea is based on the so-called Facility Location with Outliers (FLO) problem, introduced and studied in theoretical computer science in 2001 (see ref. [8] in the paper). The novelty of the proposed approach lies mainly in the study ot the Lagrangian relaxation of the original (integer) linear programming problem. In particular, the authors show that the Lagrangian relaxation is equivalent to solving the linear relaxation of FLO, and also analyze the convergence properties of the subgradient method used, which makes the algorithm highly scalable. Some experiments on both synthetic and real-world data sets (MINST) are presented.The paper is well written and organized, it’s easy to follow and understand, despite many technicalities, and its motivations and goals are quite clear.Also, I like the idea of introducing, within the machine learning community, this apparently novel formulation of the clustering problem which, to my knowledge, was originally confined only to the theoretical computer science domain. In regards to the statement made by the authors that “clustering and outlier detection are often studied as separate problems,” however, I’d like to draw the authors’ attention to novel clustering formulations which, abandoning the idea of partitioning the input data set, focus instead on the notion of a cluster itself (see, e.g., M. Pelillo, “What is a cluster?” NIPS’09 Workshop on Clustering, and references therein). In fact, in these approaches, which are non-partitional by definition, one can see that the problem of clustering and outlier detetion are simply two faces of the same coin.One problem with the proposed formulation, though, is that while it doesn’t need as input the number of clusters, which is of course good, it neverthless needs to know the number of outliers, which is even more problematic than knowing the number of clusters. Also, it needs to know the “cost” of creating a new cluster, and here, again, it’s not clear how to define them in practical applications (the choice made in the experiments reported here look quite heuristic).My final comment concerns the experimental validation. Indeed, the results presented here do not show a clear and substantial improvement over the other methods used. This is particularly clear in the MNIST experimemts (Table 1) which show that APOC and LR are basically equivalent, and k-means-- is better than the proposed LR algorithm.Minor comments:- Definition 2 is indeed a Proposition


Projective dictionary pair learning for pattern classification
Discriminative dictionary learning (DL) has been widely studied in various pattern classification problems. Most of the existing DL methods aim to learn a synthesis dictionary to represent the input signal while enforcing the representation coefficients and/or representation residual to be discriminative. However, the 0 or 1 -norm sparsity constraint on the representation coefficients adopted in most DL methods makes the training and testing phases time consuming. We propose a new discriminative DL framework, namely projective dictionary pair learning (DPL), which learns a synthesis dictionary and an analysis dictionary jointly to achieve the goal of signal representation and discrimination. Compared with conventional DL methods, the proposed DPL method can not only greatly reduce the time complexity in the training and testing phases, but also lead to very competitive accuracies in a variety of visual classification tasks.

This paper proposes a discriminative dictionary learning framework for both synthesis and analysis dictionaries, namely projective dictionary pair learning (DPL), where the analysis dictionary is learned by linear projection. The purpose of the method is to improve the computational efficiency of the coding process by discarding the explicit optimization of sparsity regularizers, e.g. l1/l0 norm without sacrificing the classification accuracy. While the paper is easy to follow there are several typos/grammatical errors/incorrect equation references, e.g. on page 4, (8) in the last sentence should be (9), that needs to be corrected. The paper deals with an topic that is of significant interest. Nevertheless, the proposed method has marginal novelty because there are related methods already existing in the literature. For instance, see the following missing but very related reference:Zhizhao Feng*, Meng Yang*, Lei Zhang, Yan Liu, and David Zhang, Joint discriminative dimensionality reduction and dictionary learning for face recognition. To Appear in Pattern Recognition. (* contribute equally to this work).In addition the reviewer has serious concerns about the proposed approach:(1) Line 191-192, bi-linear optimization problem of the objective function in Eq. (5). Why is it bi-linear?(2) Convergence. Even though the authors show an example of convergence in practice, there is no guarantee because of the normalization of D. Consequently, the convergence of the method is highly dependent on the data, limiting the scalability of the method. Is it possible to analyze the convergence theoretically?(3) Classification. The proposed algorithm optimizes the variables D, P, and A alternatively in Eq (6). However, the proposed classification rule is based on Eq (5). Why don't you still use Eq (6) as your classification rule? That is, first estimate A for a test sample using Eq (8), and then checking the reconstruction error for each class as proposed. To me this is more consistent with what you optimize, and the computational cost is the same as the proposed one if pre-computing some matrices in Eq (8).(4) Line 306, robustness to variable m. I have my doubts about this statement. My line of thinking is as follows: This parameter essentially determines the rank of the matrix DP in Eq (5) explicitly. This actually is equivalent to controlling the model complexity by essentially enforcing low-rank constraints. If we carefully check the details in the experimental section, we can find that the value of this parameter is always much smaller than the dimensionality of features used in each dataset, which guarantees the low-rank property. So why don't you plot a figure that contains the classification accuracy with increasing m?


Bandit Convex Optimization: Towards Tight Bounds
Bandit Convex Optimization (BCO) is a fundamental framework for decision making under uncertainty, which generalizes many problems from the realm of online and statistical learning. While the special case of linear cost functions is well understood, a gap on the attainable regret for BCO with nonlinear losses remains an important open question. In this paper we take a step towards understanding the best attainable regret bounds for BCO: we give an efficient and near-optimal regret algorithm for BCO with strongly-convex and smooth loss functions. In contrast to previous works on BCO that use time invariant exploration schemes, our method employs an exploration scheme that shrinks with time.

In this paper, the authors have proposed a new algorithm for Bandit Convex Optimization (BCO) with strongly-convex and smooth loss functions, which uses an exploration scheme that shrinks with time. The authors have also proved that the proposed algorithm achieves an \tilde{O}(\sqrt{T}) regret (see Theorem 10), which matches the existing best lower bound if we ignore the logarithm factors. The technical analysis of this paper looks correct.The result of this paper is significant in the sense that the authors have not only provided a tighter regret bound for BCO with strongly-convex and smooth loss functions (from \tilde{O}(T^{2/3}) to \tilde{O}(T^{1/2})), but more importantly, this improved regret bound matches the lower bound in (Shamir 2013). Thus, in general, I vote for acceptance of this paper. However, I vote for marginal acceptance instead of strong acceptance since I think the paper is not well-written, specifically,1) There are many lemmas in the paper, but when reading the paper (without looking at the appendix), it is not clear which ones are existing results and which ones are proved by the authors. This makes it a little bit hard to evaluate the technical significance of the paper.2) The authors spend too much space to review the existing results, and the main result of this paper starts at the end of page 6.3) Some symbols in the paper are not well defined, for example, \delta in Eqn(4) is not defined. Some results are not very clear, for instance, the result of Lemma 9 should hold for any \omega, but the authors do not explicitly state that.4) The authors should rewrite Algorithm 2: there should be no loop in Algorithm 2, and \Nabla h_t (x_t)'s should be inputs to the algorithm.


Orbit Regularization
We propose a general framework for regularization based on group-induced majorization. In this framework, a group is defined to act on the parameter space and an orbit is fixed; to control complexity, the model parameters are confined to the convex hull of this orbit (the orbitope). We recover several well-known regularizers as particular cases, and reveal a connection between the hyperoctahedral group and the recently proposed sorted 1 -norm. We derive the properties a group must satisfy for being amenable to optimization with conditional and projected gradient algorithms. Finally, we suggest a continuation strategy for orbit exploration, presenting simulation results for the symmetric and hyperoctahedral groups.

Summary: The authors re-explain regularization in optimization problems as a constraint of the type "the parameters ${\bf w}$ must belong to the convex set $O$" where the convex set "O" is obtained as the convex hull of all the points of the form $g.v$ where $v$ is some fix vector, $g$ an element from a group and $.$ is a (linear) group action of element $g$ on vector $v$.More concretely, their main contributions are as follows. (A) they explain how several regularizations can be obtained from their framework. For example, the ball associated to the L1 norm can be explained as the convex hull of the points obtained by flipping the sign and permuting the components of the vector $(1,0,0,..,0)$; (B) they show that given a seed $v$ and a group action associated to a group $G$, the notion of "$w$ is a member of the convex set $O_G(v)$" can be seen as "$v$ is smaller than $w$" under a pre-order; (C) they show that if $-v$ belongs to convex set $O$ then $O$ can be seen as the ball of an atomic norm (as defined in Chandra et al.); (D) they show that the L1-sorted norm equals the dual of the norm associated to the signed-pertumation orbitope; (E) they show how to reinterpret the main steps of conditional and projected gradient algorithms in the language of orbitopes and give a procedure to compute projections onto orbitopes. (F) they provide an heuristic algorithm that iteratively morphs the shape of the ball-norm associated to the regularizer, generalizing the idea of regularization paths.Quality: There are no technical mistakes in the paper. The idea of morphing the shape of the regularizer's ball-norm is the most interesting idea in my opinion. In this regard, it would be good if the authors could clarify the following. Homotopy methods build complete regularization paths that, after being computed, are used in combination with, for example, cross-validation to find the right amount of regularization to perform. I do not understand why the continuation algorithm stops "at (the) point regularization is not having any effect". It would also appreciate if the authors could say a few words about how the continuation algorithm would performs when $\epsilon = 0$. In other words, the shape of the ball-norm is changed by its size is kept constant. Does the algorithm converge ?Proposition 10 has a trivial pictorial explanation that might be good to include for the sake of clarity. In particular, taking the dual of the norm associated to singed-permutations corresponds to transforming the edges in the ball-norm of Fig. 1-right to vertices and transforming vertices to edges. This leads immediately to the ball-norm of the sorted L1-norm, that can be seen as the intersection of the ball-norms of all weighted L1-norms obtained by permuting the coefficients $w$.In Prop. 3 the authors show that (under some conditions) orbit regularizers can be seen as atomic norms. It would be good to explain when/how atomic norms can be seen as orbit regularizers. Clarity: The paper is overall very well written and clear. Here are a few minor things that can be improved. In Line 071 there is a parenthesis missing. In line 244-246  subscripts are missing in $m({\bf w},{\bf v})$, having them would be better. The quality of the pictures should be improved. Are pictures vector format? When I print the paper they look blurred. It would be very useful to have numbers in the references, [1], [2], etc. In Fig. 4, what is the scale in the y-axis referring too?Significance: I find the idea of morphing the shape of the regularizer's ball-norm potentially interesting. Unfortunately, the numerical results do not clearly show that the continuation algorithm, as is, leads to significantly better performance. In Fig. 5 only one simple example is analyzed. Also, the results from Fig. 4 seem inconclusive. It would be good to report the number of iterations it takes for the continuation algorithm to converge. Originality: The idea of re-explaining regularization using orbitopes is new. The idea of a continuation algorithm that iteratively morphs the shape of ball-norms in addition to scaling them is, as far as I can tell, new.


On Sparse Gaussian Chain Graph Models
In this paper, we address the problem of learning the structure of Gaussian chain graph models in a high-dimensional space. Chain graph models are generalizations of undirected and directed graphical models that contain a mixed set of directed and undirected edges. While the problem of sparse structure learning has been studied extensively for Gaussian graphical models and more recently for conditional Gaussian graphical models (CGGMs), there has been little previous work on the structure recovery of Gaussian chain graph models. We consider linear regression models and a re-parameterization of the linear regression models using CGGMs as building blocks of chain graph models. We argue that when the goal is to recover model structures, there are many advantages of using CGGMs as chain component models over linear regression models, including convexity of the optimization problem, computational efficiency, recovery of structured sparsity, and ability to leverage the model structure for semi-supervised learning. We demonstrate our approach on simulated and genomic datasets.

This paper explores how sparse conditional gaussian graphical models (sparse CGGMs) can be used as components of chain graphs. The authors discuss the qualitative and quantitative advantages of using CGGM formulation compared to using sparse multivariate regression with covariance estimation (MRCE) for two and three layer Gaussian chain graph. They go further to show how the framework can easily be extended to handle missing data in the middle layer as semi supervised learning. On their simulation data they show clear advantage of making use of the using CGGM for both network structure reconstruction and for prediction accuracy. Pros:The paper is clearly written. It shows how sparse CGGMs can be embedded in a multi layer chain graph structure and covers both structure recovery and inference in the resulting models. The models have intuitive interpretation in the context of genomic data. The improvement in both structure learning and prediction compared to using linear regression components are clear in the synthetic datasets considered.Cons: In general, the authors build heavily on previous works that developed sparse CGGMs (Shon & Kim 2012, Wytock & Kolter 2013). The review style of writing makes it hard to pin down directly from the text what was previously done and what is novel. The authors should state clearly the contributions.The results/evaluation for real life genomic data evaluation appear limited, cryptic and unconvincing. This makes the method overall less convincing especially as a way to tackle the bio-medical analysis problem they pose. (a) How are the dimensions/features selected? The numbers (e.g. 1000 SNPS, 200 expression "traits"?) seem unrealistic/irrelevant for the general task they offer to tackle originally (millions of SNPS, thousands of genes etc.)  (b) How is performance evaluated and what is the meaning of the prediction error values in Table 1? How is "structure" being scored in this setting?? (c) There are no error bars for these results (Table 1) (d) Contrary to the claim in the main text (Line 422) the improvement over LR models appears to be minute to non existing. The authors comment about the ability to deduce dependency between input features (p. 5 top) but this ability is not explored in their experimental evaluations.Other comments:Line 063: "more appropriate" than what?Line 74-75: "We show that our approach  can reveal the module structure ...in different types of genomic data effectively" - this is actually shown only for synthetic data experiments. The relation to real genomic data is not clear.Line 158: "recently it has been argued" Where? please give a citation.Line 204: Many of the multi task learning works do not assume the different "tasks" or outputs share the same values from the input features, which is the underlying assumption in the CGGM model. The authors should clarify that point to put their work in the right context.Figure 2: It seems x6 is irrelevant for the points been made in both Fig2a and 2b. If so consider removing it.Line 261: "It is well-known that...." is rather a strong statement that implies the relations described always hold. Consider revising (e.g."in many cases...") and adding an appropriate reference(s) that clearly establish the soundness of the claim.


Transportability from Multiple Environments with Limited Experiments: Completeness Results
This paper addresses the problem of mz-transportability, that is, transferring causal knowledge collected in several heterogeneous domains to a target domain in which only passive observations and limited experimental data can be collected. The paper first establishes a necessary and sufficient condition for deciding the feasibility of mz-transportability, i.e., whether causal effects in the target domain are estimable from the information available. It further proves that a previously established algorithm for computing transport formula is in fact complete, that is, failure of the algorithm implies non-existence of a transport formula. Finally, the paper shows that the do-calculus is complete for the mz-transportability class.

The authors address the problem (called mz-transportability) of inferring a causal relationship R in a target domain using causal knowledge obtained in multiple source domains with limited experiments.In the target domain only passive observations are available (or also some limited causal knowledge) which is not enough alone for the inference of R.Specifically the contributions are:1. A necessary and sufficient condition for deciding when causal effects in the target domain are estimable from the available information.2. A proof that a previously proposed algorithm (TR^mz) for computing the transport formula is complete.3. A proof that the do-calculus is complete for establishing mz-transportability of causal effects.*Quality & Clarity:This is mainly a theoretical paper and the claims are well-supported by the authors. It is well-organized and well-presented. However, the claims and proofs are often quite involved (as also pointed out by the authors) and I found it quite difficult to follow some parts (for example page 6) and fully understand the details. Nevertheless, I would probably attribute this more to the advanced theoretical analysis rather than to the lack of good presentation of the paper. I would propose for improvement:- Define what is "a maximal root set" in Definition 4 (like in [21]). - In Def.5: "(called an hedge)" -> "(called an hedge for P_x(y))".- The definition of an hedge after Definition 4 is confusing. You could include a more rigorous definition of what is an hedge there, instead of including it in Definition 5.- What do you mean by "separable"?- It would be beneficial for the reader to present some parts in a simpler way, if possible.*Originality & Significance:The paper builds on the NIPS 2013 paper ("[1] Transportability from Multiple Environments with Limited Experiments") which is adequately referenced. [1] included necessary OR sufficient conditions for transportability and proved only soundness of TR^mz. The current paper advances over [1] by providing conditions for complete characterization of transportability, proving that TR^mz is complete and proving that do-calculus is complete for the mz-transportability class. So, I consider the paper having a significant contribution. To be more explicit, you could add in the beginning of section 3 that the first part of this section (page 4) repeats from [1] or other publications (not only referring to them in page 4). Also, for eq. (2), add a reference to [1], where it is explain how eq. (2) is acquired.Some more comments:- Abstract, 3rd line: it seems that "limited experimental data" only refers to the target domain and not to the sources. Consider changing appropriately.- Motivation, 1st line: maybe missing a "that" after "experiments"? Consider rewriting sentence.- p.2, par.4: "the goal of....conditions the causal effect" -> "...conditions causal effects in the target domain"- First bullet before section 2: last line: "in the domains" -> "in the source domains"?- Section 2, line 2: you refer to the node S of Fig.1(a) which is missing: Consider adding the letter S in this square node of the figure.- Footnote 1: "..exclusively, on.." -> "..exclusively, of.."- Footnote 1: along with refs [13,14] you could also consider some of the following references:Ben-David et al. Analysis of representations for domain adaptationMansour et al. Domain adaptation: learning bounds and algorithmsMansour et al. Domain adaptation with multiple sourcesBen-David et al. A theory of learning from different domains- p.3, last line of 1st par: period before "our goal is"- explain what is V, U etc. in M_x- what is "IS" after Definition 2?- After Theorem 1: "It is not difficult to see..": add reference to [1]- middle p.5: period before "the result follows"- first line after Definition 4: remove "and" before "rooted"- p.7 middle: period before "refer to Appendix 2.."- Theorem 4:, last line: "contain" ->"contains", "spans"->"span", "a" ->"an" and "F'". What are sC-forests?- proof of Thm 4: don't you need capital C for c-forest?,   "shows" -> "showed"- After Thm 4 write that the proof is provided in the supp. material.- Maybe it would be better to use pi_c instead of pi_a for the source of Fig.1(c).- Supplementary, last page:   -- proof of Thm 5: remove one "the" (first line), remove one full stop (least line)  -- Proof of Corollary 1: "...from the previous Corollary": do you mean from the previous Theorem?


Factoring Variations in Natural Images with Deep Gaussian Mixture Models
Generative models can be seen as the swiss army knives of machine learning, as many problems can be written probabilistically in terms of the distribution of the data, including prediction, reconstruction, imputation and simulation. One of the most promising directions for unsupervised learning may lie in Deep Learning methods, given their success in supervised learning. However, one of the current problems with deep unsupervised learning methods, is that they often are harder to scale. As a result there are some easier, more scalable shallow methods, such as the Gaussian Mixture Model and the Student-t Mixture Model, that remain surprisingly competitive. In this paper we propose a new scalable deep generative model for images, called the Deep Gaussian Mixture Model, that is a straightforward but powerful generalization of GMMs to multiple layers. The parametrization of a Deep GMM allows it to efficiently capture products of variations in natural images. We propose a new EM-based algorithm that scales well to large datasets, and we show that both the Expectation and the Maximization steps can easily be distributed over multiple machines. In our density estimation experiments we show that deeper GMM architectures generalize better than more shallow ones, with results in the same ballpark as the state of the art.

This paper introduces the idea of deep Gaussian mixture models.  A GMM can be seen as consisting of a single isotropic unit norm Gaussian, where each of the components of the mixture consists of applying a different linear transformation to that Gaussian.  This idea is extended to the case of a multilayer network, where each node in the network corresponds to a linear transformation, and each route through the network corresponds to a sequence of linear transformations.  The number of mixture components is then the number of routes through the network.This is a clearly written paper.  The idea is both simple and good.  Overall I liked the paper.  I did think that the experimental results were disappointingly low dimensional, especially considering how much effort was spent in the paper discussing how to make the algorithm computationally tractable.Also, I think the proposed algorithm is quite closely related toTang, Yichuan, Ruslan Salakhutdinov, and Geoffrey Hinton. "Deep mixtures of factor analysers." International Conference on Machine Learning (2012).this probably deserves a citation and discussion.I would strongly encourage the authors to release the source code for the experimental results as supplemental material.  This makes the paper more convincing, increases citation counts, and encourages a culture of reproducible science which benefits everyone.More detailed comments follow:39 - "in function" -> "as a function"73 - "A_3,1" -> "A_1,3"89 - You provide good arguments in 312-318 for why this is true for your algorithm.  I'm skeptical that EM is more parallelizable than stochastic gradient descent in general though.  Maybe phrase this more specifically.139 - There are an exponential number of paths through the network.  Assigning a probability to each of them without factorizing them would become intractable quite quickly -- even for relatively small networks.  I'm surprised this factorization isn't needed almost all the time.199 - This is commonly called the MAP approximation.  Why not instead just sample a single nonzero gamma with probability proportional to pi (this is EM, with the E step represented as a sample from the posterior).  Unlike when using the MAP approximation, this will give you an unbiased estimate of your learning gradient, and should lead to a higher log likelihood model.230 - "in function of" -> "as a function of"235 - In the plot it never converges to the optimum.251 - "and scalable." -> "and in a scalable way."307, 341 - If you want to do quasi-Newton optimization with minibatches, and without hyperparameters, you might try https://github.com/Sohl-Dickstein/Sum-of-Functions-Optimizer312-318 - Cool.  This provides good support for the benefits of EM.331 - "have to construction" -> "have to construct"337 - with using -> using344 - I strongly suspect this is a side effect of the MAP approximation to the posterior, as opposed to not optimizing fully during the M step.  (As a note -- EM can be seen as maximizing a lower bound on the log likelihood.  Importantly, that lower bound increases even if the M step is not run to convergence.)Figure 4 - I don't think this X-axis makes much sense for the Deep GMMs -- there's nothing special about the number of components in the top layer.  Number of parameters would probably make a better x-axis.436 - Author name listed as "anonymous".


Smoothed Gradients for Stochastic Variational Inference
Stochastic variational inference (SVI) lets us scale up Bayesian computation to massive data. It uses stochastic optimization to fit a variational distribution, following easy-to-compute noisy natural gradients. As with most traditional stochastic optimization methods, SVI takes precautions to use unbiased stochastic gradients whose expectations are equal to the true gradients. In this paper, we explore the idea of following biased stochastic gradients in SVI. Our method replaces the natural gradient with a similarly constructed vector that uses a fixed-window moving average of some of its previous terms. We will demonstrate the many advantages of this technique. First, its computational cost is the same as for SVI and storage requirements only multiply by a constant factor. Second, it enjoys significant variance reduction over the unbiased estimates, smaller bias than averaged gradients, and leads to smaller mean-squared error against the full gradient. We test our method on latent Dirichlet allocation with three large corpora.

OverviewThe authors proposed a new approach of smoothing gradients for stochastic variational inference by averaging the sufficient statistics across a fixed time window. With the proposed method and a proper chosen time window size, the variance of stochastic gradients can be reduced, and the convergence speed of the algorithm can be improved. However, the sufficient statistics averaging introduce a bias to the gradients, which might lead to a less optimal solution. QualityThis work proposes a gradient smoothing scheme via averaging the sufficient statistics. The gradient smoothing reduces variance of gradients, while paying the price by introducing a bias into gradients. It is an interesting idea to average the sufficient statistics instead of averaging gradients which has been studied before. The experiments with LDA show a good result empirically. It has been shown that the gradient smoothing scheme can reduce variance of gradients and improve model performance with a proper time-window size. The choice of the time-window size becomes a tuning parameter for the tradeoff between bias and variance. In general, introducing a small bias to the model is not a big problem, since variational inference approaches introduce an intrinsic bias while optimizing the evidence lower bound.I have several reservations regarding this work. Different from introducing a bias into the objective function of model, the biases in gradients might be accumulated throughout learning, which potentially lead to a solution very far from local optima. It would be interesting to study the properties of the bias term, and see the new objective function that the bias term leads to. As mentioned in the discussion section, the convergence of the proposed method has not been proved. Some demonstration of convergence, at least empirically, needs to be shown. According to the experimental results, the choice of the time window size is crucially to the performance of the method. It would be useful to discuss the principal of choosing the right window size.ClarityThis paper is, in general, well written, and easily understandable. The equation reference on line 240 is missing. In figure 3, the results of SVI are not converged. Please show the results until convergence. It will be also good to show the performance of original LDA as a baseline, to have an idea of the performance difference between SVI-based approaches and original LDA for general audience.OriginalityThis work looks like an original work with an interesting idea and good performance.SignificanceThis work presents a new gradient smoothing scheme for stochastic variational inference. It would potentially be very useful for the variational inference community.


Generalized Higher-Order Orthogonal Iteration for Tensor Decomposition and Completion
Low-rank tensor estimation has been frequently applied in many real-world problems. Despite successful applications, existing Schatten 1-norm minimization (SNM) methods may become very slow or even not applicable for large-scale problems. To address this difficulty, we therefore propose an efficient and scalable core tensor Schatten 1-norm minimization method for simultaneous tensor decomposition and completion, with a much lower computational complexity. We first induce the equivalence relation of Schatten 1-norm of a low-rank tensor and its core tensor. Then the Schatten 1-norm of the core tensor is used to replace that of the whole tensor, which leads to a much smaller-scale matrix SNM problem. Finally, an efficient algorithm with a rank-increasing scheme is developed to solve the proposed problem with a convergence guarantee. Extensive experimental results show that our method is usually more accurate than the state-of-the-art methods, and is orders of magnitude faster.

Summary: this paper presents a method to decompose and complete tensors. The main idea is to represent the tensor as a product of a low-rank core tensor and a set of orthonormal matrices, and then to minimise the nuclear norm of the core tensor (that is, the sum of the nuclear norms of its unfoldings). Intuitively the algorithm mixes fixed rank decomposition (using an upper bound on the rank) with norm-regularisation. The approach is analysed theoretically (in terms of convergence & recovery) and empirically (on one synthetic dataset and one real-world dataset). Quality: This is a nice idea, and I think it has been executed well. The optimisation is non-trivial and comes with some guarantees, and the scalability of the algorithm is convincing. Results suggest good performance on the chosen dataset compared to a set of state-of-the-art methods. There are some things that concern me. First, I find the empirical comparison relatively weak as only a single dataset was considered, and this dataset seem to have been used in other tensor factorisation papers (authors, please clarify if this is not the case and the data has indeed been used before). Why not use datasets from other papers, such as the network traffic data of Acar et al, or the Ocean video of Liu et al? Second, I find the stability guarantees not very convincing. It is true that the middle term in equation 21 diminishes with increasing sample size and hence the impact of the constant C, but without any bounds on C to me this theorem does provide much practical use (that said, Wang & Xu give the same type of guarantee). However, I find the method useful enough to not count this against the paper, it’s an add-on to me. Question: I am a little surprised that your method is faster than the WTucker method, as my impression is that your essentially solve something like a Tucker decomposition in your inner loop (when estimating your U and G matrices/tensors). Am I missing something?  Clarity: Generally I found this paper to be quite well written. However, I think the description of the algorithm could be slightly improved by making it the currently used set of ranks more explicit. The algorithm is maintaining one rank per mode, which is increased over time (to always reach the R_i, … R_N ranks). If you mention this before equation 9, and that the current ranks are used in this equation (not the R_i chosen in advance), it would be much clearer to me. Second, it would be helpful to discuss briefly implications (and interpretations) of section 4. For example, why could Theorem 4 be helpful even though you have no bound on C?  Originality: I find the main idea relatively original. It brings together a few ideas in a novel way (norm of core tensor = norm of full tensor, ADMM, HOOI, stability analysis a la Wang & Xu). It’s still not gonna change the way of thinking of the field.  Significance: While the method itself isn’t ground breaking, the outcome of increased scalability w/ less sensitivity to rank estimation is quite significant. This could certainly be a method I’d be eager to try (although I’d still like to see more empirical evidence that this works).


Augur: Data-Parallel Probabilistic Modeling
Implementing inference procedures for each new probabilistic model is timeconsuming and error-prone. Probabilistic programming addresses this problem by allowing a user to specify the model and then automatically generating the inference procedure. To make this practical it is important to generate high performance inference code. In turn, on modern architectures, high performance requires parallel execution. In this paper we present Augur, a probabilistic modeling language and compiler for Bayesian networks designed to make effective use of data-parallel architectures such as GPUs. We show that the compiler can generate data-parallel inference code scalable to thousands of GPU cores by making use of the conditional independence relationships in the Bayesian network.

This paper introduces a new probabilistic programming language for Bayesian networks, Augur, and a compiler that are designed to produce parallel inference with GPU. The whole system is composed of two parts, the front end, which converts the model description embedded in Scala into a symbolic representation, and the back end, which generates CUDA code from the intermediate representation at runtime. Compared to other probabilistic languages, Augur is different in that it is designed to produce parallel inference method for scalability, and that it uses a compact symbolic representation for distributions.It is a meaningful contribution to automatically produce a parallel inference program for a general Bayesian model. Also, the LDA experiments addresses a common scalability problem in probabilistic programming that much more memory would be used in a probabilistic program than a hand-written program. I find it not difficult to understand the main idea of this paper, although a lot of details such as the construction of IR and the inference configuration in Augur are omitted.My main concern is about the efficiency and the overhead of the automatically generated CUDA code:First, it is not trivial to write a CUDA program to achieve high parallelization given a problem, and the efficiency depends highly on the number of independent tasks and the memory usage. The current proposed method seems to simply take a product expression in the symbolic representation and parallelize the computation on GPU. Is there a general mechanism for the compiler to detect which product should be parallelized and which should be rather run sequentially on CPU? Second, the authors discusses two algorithms to optimize the sampling of multiple Dirichlet distributions, and the compiler will take the better one according to runtime values.  Does this strategy require extra work to apply to a new distribution, or can you make it a general rule for all similar models?Lastly, AFAIU the CUDA code is generated once at the beginning of a MAP/Sampling call with a constant time cost, and the currently values of the model variables is used to optimize the generated code. However, values such as the number of subtasks for a product operation can depend on other variables and is changed during the execution. Would that mean a recompilation of the CUDA code? Another concern is the limited expressive capability of Augur. The currently language does not allow a variable size for the model support. So it can not be used for many model selection problems, e.g., a Gaussian mixture model with the number of clusters being a random variable with a prior distribution. Can author show that Augur supports all the Bayesian networks?The experiment results for the multivariate linear regression is not impressive. The parallelized code by Augur performs similarly to the sequential code by Jags. Also, the labels and plots in Figure 2(a) are unreadable. For the LDA experiments, it is well known that the collapsed sampler converges faster than the uncollapsed sampler as a function of the sample size. However, I am suprised that Figure 2(b) shows that the Factorie converges slower and also to a significantly lower value of the predictive probability than the other methods.--------------Updates after feedback:The authors feedback addresses my questions on the efficiency of the CUDA code to my satisfaction. However, their response on Factorie confirms my concern on the comparison between CUDA parallelized sampler and the collapsed Gibbs sampler. The plot of Factorie does not show the actual performance of collapsed Gibbs on the predictive probability. It is therefore hard to judge if Augur achieves the same performance as collapsed Gibbs given the same amount of time. I suggest the authors discusses this issue explicitly in a final version of the paper.


Object Localization based on Structural SVM using Privileged Information
We propose a structured prediction algorithm for object localization based on Support Vector Machines (SVMs) using privileged information. Privileged information provides useful high-level knowledge for image understanding and facilitates learning a reliable model even with a small number of training examples. In our setting, we assume that such information is available only at training time since it may be difficult to obtain from visual data accurately without human supervision. Our goal is to improve performance by incorporating privileged information into ordinary learning framework and adjusting model parameters for better generalization. We tackle object localization problem based on a novel structural SVM using privileged information, where an alternating loss-augmented inference procedure is employed to handle the term in the objective function corresponding to privileged information. We apply the proposed algorithm to the Caltech-UCSD Birds 200-2011 dataset, and obtain encouraging results suggesting further investigation into the benefit of privileged information in structured prediction.

- The method is a fairly direct generalization of the framework of learning with privileged information to the structured output case. The method is effective for the object localization task and results in good improvements in localization accuracy. - Does this formulation of SSVM+ reduce to SVM+ for binary classification with the right choice of loss function and feature vector, just as SSVM reduce to SVM in the classical case? It looks like the authors' formulation of SSVM+ contains separate slack variables \xi_i for each example x_i and there are extra degrees of freedom. - The loss functions \Delta and \Delta^* seems to be the same in the object localization task, but it is not stated clearly in section 3.3. - How effective is the alternating inference procedure between y and y^*? How many alternating iterations are required? When the parameter vectors w and w^* are far from the optimal solution, could this alternating inference procedure get stuck in bad local minima? - I am interested in knowing the strength of the privileged information encoded in the object localization task. If we train a regular SSVM with the privileged information and also use the privileged information during testing, how well will the algorithm do? This can give us some understanding on how informative the privileged features are when they're available during test time, vs only during training time under the LUPI framework. - Overall I think the method is a simple extension of the learning under privileged information framework, but it works well. It can be improved if the authors can give more explanations of their formulation and how it links to the case of binary SVM+, or give some theoretical justification like those under the LUPI framework.


Articulated Pose Estimation by a Graphical Model with Image Dependent Pairwise Relations
We present a method for estimating articulated human pose from a single static image based on a graphical model with novel pairwise relations that make adaptive use of local image measurements. More precisely, we specify a graphical model for human pose which exploits the fact the local image measurements can be used both to detect parts (or joints) and also to predict the spatial relationships between them (Image Dependent Pairwise Relations). These spatial relationships are represented by a mixture model. We use Deep Convolutional Neural Networks (DCNNs) to learn conditional probabilities for the presence of parts and their spatial relationships within image patches. Hence our model combines the representational flexibility of graphical models with the efficiency and statistical power of DCNNs. Our method significantly outperforms the state of the art methods on the LSP and FLIC datasets and also performs very well on the Buffy dataset without any training.

This work proposes a method for monocular human pose estimation. Theproposed method is based on a tree-structured graphical model withimage dependent pairwise relationships. These relationships arerepresented by a mixture model and local image measurements are usedto predict a mixture component for each pairwise term. Prediction isbased on features extracted from image patches using DeepConvolutional Neural Networks (DCNN). DCNN are trained to predictconditional probabilities for 1) spatial relations between body partsand 2) presence of individual body parts. Finally, mixture weightsunary and pairwise terms are jointly learned in a max-marginframework.This is an interesting paper with several strong points:- significance. This paper addresses a challenging and highly research problem of  monocular human pose estimation and thus it may have an impact on  significant part of community- model. Using deep learning features for local conditioning of  pairwise terms in a graphical model is an interesting idea.- experimental evaluation. The work provides some insights by breaking  down the performance of individual model components. The importance  of image conditioned pairwise terms for pose estimation performance  is evident.- results. This work shows promising results significantly  improving over state of the art on standard pose estimation benchmarks.However, this paper also has several weak points:- conditioning of pairwise terms in a graphical model for human pose  estimation has been done before by [SJT10] (see citation below) and  [11]. I struggle to see the principal differences between this  method and the previous work, except that this paper uses  stronger image features and local conditioning. Moreover, it is not  clear that local conditioning of pairwise terms has more advantages  over global conditioning, as local conditioning allows to model  pairwise relationships between adjacent body parts *only*, which is  in contrast to the previous method [11] which models longer range  part dependencies. - the proposed method heavily builds on a Yang&Ramanan (Y&R) model  [20,21] and reads like an extension of Y&R model with  image-conditioned pairwise terms.- choosing a number of mixture components for the pairwise terms (11)  is not justified. Y&R model [21] uses 6 components only. Fairer  comparison between two models would be when the number of mixture  components is the same.- using 0-1 loss during training is suboptimal, as at test time  different types of losses measuring the fraction of correctly  predicted body parts/joints are used. A more proper way would be to  use the same loss during testing and training.- comparison to Deep Pose [17] on LSP dataset is not completely fair,  as [17] report results using person-centric annotations, which is a  harder task.- some citations are missing[SJT10]  B. Sapp, C. Jordan, and B. Taskar. Adaptive pose priors for     	 pictorial structures. In CVPR’10. [KU12]	 L. Karlinsky, and U. Shimon. Using linking features in learning     	 non-parametric part models. In ECCV'12- unrealistic poses resulting from data augmentation by rotating each  positive training image through 360 with a step of 10 degrees may  hurt the performance. The paper advocates this procedure by saying  that the patches extracted from the unrealistically rotated poses  still contain information about local part relations. However, it  has been suggested [20,21] that the appearance of body parts is  rotation dependent, and thus is characteristic for particular part  orientations. Moreover, the local patches seem to be large enough to  include a significant portion of the context, e.g. supporting  surfaces for legs in upright or for arms in upside down positions,  which may enhance discriminative power.Minor comments:- saying that the method "performs very well on Buddy dataset without  any training" is a bit misleading. What is actually meant is that  the method was not trained on Buffy dataset, but trained on other  datasets and tested on Buffy.- what is the training/testing time of the method?


Incremental Local Gaussian Regression
Locally weighted regression (LWR) was created as a nonparametric method that can approximate a wide range of functions, is computationally efficient, and can learn continually from very large amounts of incrementally collected data. As an interesting feature, LWR can regress on non-stationary functions, a beneficial property, for instance, in control problems. However, it does not provide a proper generative model for function values, and existing algorithms have a variety of manual tuning parameters that strongly influence bias, variance and learning speed of the results. Gaussian (process) regression, on the other hand, does provide a generative model with rather black-box automatic parameter tuning, but it has higher computational cost, especially for big data sets and if a non-stationary model is required. In this paper, we suggest a path from Gaussian (process) regression to locally weighted regression, where we retain the best of both approaches. Using a localizing function basis and approximate inference techniques, we build a Gaussian (process) regression algorithm of increasingly local nature and similar computational complexity to LWR. Empirical evaluations are performed on several synthetic and real robot datasets of increasing complexity and (big) data scale, and demonstrate that we consistently achieve on par or superior performance compared to current state-of-the-art methods while retaining a principled approach to fast incremental regression with minimal manual tuning parameters.

The authors propose a regression scheme based on locally weighted regression (LWR). By making links from LWR to Gaussian process (GP) regression, a probabilistic model is formed. The model initially appears to have scalability like a GP (cubic in N) but a variational scheme rescues the method. The updates in the variational scheme mimic the LWR procedure, recovering scalability of the algorithm. Clarity--The paper is very well written. It's easy to follow, the notation is simple and consistent, and sufficient detail is presented without being overwhelming. The link from LWR to GP regression is especially neat. Although it's clear that any individual model in the LWR scheme comprises a GP (as any linear Gaussian model), it seesm novel to interpret the entire methodology as such. If one section of the paper is unclear, it is section 4. The authors may protest constraints on space, but this section seems somewhat anecdotal in comparison to the thoroughness of the rest of the paper. The presentation of the Algorithm is a help, I suppose. Quality and significance--The paper is technically sound, is of interest to a large section of the NIPS community, and contains solid experiments. The experiments chosen represent interesting challenges and the proposal appears to make a good improvement in terms of speed whilst maintaining accuracy. I would really like to see a mention of the availability of the implementation, which would enhance the paper further. I have one quality related complaint: the probabilistic nature of the algorithm is not explored in the experiments. I would have like to have seen the average log-density of held -out data alongside the MSE. Whilst the LWR method might not provide probabilistic estimates, the proposed method and the SSGPR will. Surely in a robotics environment, where decisions have to be made under uncertainty, log p(y*) is  a more informative measure than MSE? It is widely known that different GP approximations perform very differently in terms of predictive density (e.g. FITC usually provides conservative predictive density): perhaps the authors could provide a supplementary table with the log density scores? Queries. --To make the variational approximation tractable, you have to introduce uncertainties via the parameters beta. In practise, what did these converge to ? Does this slight change of model have a strong impact?The variational updates for the local models are independent, but the beta parameters are global: does this make the mode computationally costly? Do you interlace fewer of these updates with the local updates?Table 2: The SSGPR method was pre-trained with 200 features, but the LWGPR method was allowed to use around 500 local models. Would it be fair to say that both models are of the same complexity? Does the SSGPR method not do better with more that 200 features: I seem to recall that the method scales cubically in the number of features, so could you not have afforded a few more? In table 3 the discrepancy is more severe, I guess due to the offline-online differences in the procedures.Summary--A very well presented paper, enjoyable to read. I have a few technical questions, and my overall score may change depending on the authors rebuttals.


Encoding High Dimensional Local Features by Sparse Coding Based Fisher Vectors
Deriving from the gradient vector of a generative model of local features, Fisher vector coding (FVC) has been identified as an effective coding method for image classification. Most, if not all, FVC implementations employ the Gaussian mixture model (GMM) to characterize the generation process of local features. This choice has shown to be sufficient for traditional low dimensional local features, e.g., SIFT; and typically, good performance can be achieved with only a few hundred Gaussian distributions. However, the same number of Gaussians is insufficient to model the feature space spanned by higher dimensional local features, which have become popular recently. In order to improve the modeling capacity for high dimensional features, it turns out to be inefficient and computationally impractical to simply increase the number of Gaussians. In this paper, we propose a model in which each local feature is drawn from a Gaussian distribution whose mean vector is sampled from a subspace. With certain approximation, this model can be converted to a sparse coding procedure and the learning/inference problems can be readily solved by standard sparse coding methods. By calculating the gradient vector of the proposed model, we derive a new fisher vector encoding strategy, termed Sparse Coding based Fisher Vector Coding (SCFVC). Moreover, we adopt the recently developed Deep Convolutional Neural Network (CNN) descriptor as a high dimensional local feature and implement image classification with the proposed SCFVC. Our experimental evaluations demonstrate that our method not only significantly outperforms the traditional GMM based Fisher vector encoding but also achieves the state-ofthe-art performance in generic object recognition, indoor scene, and fine-grained image classification problems.

This paper proposes a method for fisher vector coding (FVC) of high dimensional local features. Most existing approaches that use FVC assume that local features are drawn from a Gaussian Mixture Model (GMM) where the number of mixture components is specified a priori. This paper argues that while this has worked well for FVC of low-dimensional local features like SIFT, it does not scale to high dimensional features because the number of Gaussians required to model the distribution of local features becomes prohibitive. Instead of using a GMM with a finite number of mixture components, this paper proposes to approximate an infinite Gaussian mixture using a set of basis vectors. Under the assumption of iid feature dimensions, the basis vectors and feature encoding can be learned using standard sparse coding. The paper provides derivation of the corresponding fisher vector, which can be computed analytically.The paper is well-written. There are minor typos and grammatical errors throughout the paper, but they do not significantly impact comprehensibility. The proposed approach is technically sound and appears to yield good results empirically. One limitation of the approach that the paper glosses over is the assumption of a diagonal covariance matrix with constant sigma in deriving the log-likelihood in eq. 4. This is an important theoretical limitation, as GMMs can model correlation between feature dimensions, while the sparse coding based approach proposed in this paper does not. The authors should mention this limitation, and explicitly state that using PCA to first transform the local features is a practical method to get around this problem.Experiments demonstrate that the proposed approach outperforms the use of GMM-based FVC and achieves state-of-the-art performance on several image classification, scene classification, and fine-grained image classification datasets. The paper investigates the tradeoff between increasing the number of mixture components versus using higher dimensional local features for GMM-based FVC. However, it would be even more informative if the paper could also provide a similar investigation on the tradeoff between the number of basis vectors and the dimensionality of local features for the proposed sparse coding FVC.Overall, the paper contains good algorithmic novelty and demonstrates convincing and extensive empirical results. Hence, I would recommend that it be accepted for publication if the authors could provide an analysis of the tradeoff between the number of basis vectors and the dimensionality of local features for the proposed sparse coding FVC.Minor Questions:- For Pascal VOC 2007 results, there is a huge performance difference between FVC-based methods and CNN-SVM for bottle (in CNN-SVM's favor) and bus (in FVC's favor). Have you looked into the cause of these differences?Typos:- line 151: By "assuming" ...- line 240: ... coding value u_k "multiplied by" the reconstruction residue ...


Iterative Neural Autoregressive Distribution Estimator (NADE-k)
Training of the neural autoregressive density estimator (NADE) can be viewed as doing one step of probabilistic inference on missing values in data. We propose a new model that extends this inference scheme to multiple steps, arguing that it is easier to learn to improve a reconstruction in k steps rather than to learn to reconstruct in a single inference step. The proposed model is an unsupervised building block for deep learning that combines the desirable properties of NADE and multi-prediction training: (1) Its test likelihood can be computed analytically, (2) it is easy to generate independent samples from it, and (3) it uses an inference engine that is a superset of variational inference for Boltzmann machines. The proposed NADE-k is competitive with the state-of-the-art in density estimation on the two datasets tested.

The paper describes a new variant of (order agnostic, deep) NADE called NADE-k that performs k successive reconstructions/inference steps when imputing the unobserved components of the input vector. To train NADE-k, one must sample a subset of the input variables to impute while observing the remaining input variables. The paper includes experiments showing improved log probs over other versions of NADE on binarized MNIST digit generation and Caltech-101 silhouette generation. There are also experiments exploring different values of k and the variance of the imputations with respect to the choice of variable order.The NADE-k algorithm presented in the paper seems sensible and is for the most part explained clearly. The experiments included are reasonable ones. The NADE-k algorithm is an incremental improvement of deep, order-agnostic NADE and a relatively straightforward one. However, it is more expensive at both training time and test time (since test k is best set to equal training k). Any algorithm based on unrolling inference steps and backpropagating through them can have a k step version of itself created. The question is really how interesting a multi-step variant is and whether it is worth the computational expense. If CD-1 had been introduced without reference to the infinite steps of maximum likelihood and separately from CD-k, would a paper on extending CD-1 to CD-k have been very novel?One possible experiment not included in the paper that would have been interesting to see is whether slowly increasing k during training works well. In RBM training, to get very good generative performance quickly, it is helpful to switch from CD-1 to CD-k by stepping up k during training. At the start of training, the model is so inaccurate that CD-1 can make rapid progress, but as training continues, more CD steps can help. Would this idea benefit NADE-k?Somewhat more practical applications of high quality density models would have been welcome to motivate the paper more strongly, perhaps in compression or denoising. Learning a density over binarized MNIST pixels is not a very compelling task. Since the NADE family of algorithms provide tractable, normalized, high-quality density models, an application where those features are essential would have been more interesting.The paper was mostly clear. There are a few sentence-level issues, such as line 199 "allow to compute" and footnote 2 "for helping better converge."I had to slow down a lot while reading when I came upon equation 10, it would be clearer if there was a bit more english text introducing it. In general, the paper is well organized.


Multi-scale Graphical Models for Spatio-Temporal Processes
Learning the dependency structure between spatially distributed observations of a spatio-temporal process is an important problem in many fields such as geology, geophysics, atmospheric sciences, oceanography, etc. . However, estimation of such systems is complicated by the fact that they exhibit dynamics at multiple scales of space and time arising due to a combination of diffusion and convection/advection [17]. As we show, time-series graphical models based on vector auto-regressive processes [18] are inefficient in capturing such multi-scale structure. In this paper, we present a hierarchical graphical model with physically derived priors that better represents the multi-scale character of these dynamical systems. We also propose algorithms to efficiently estimate the interaction structure from data. We demonstrate results on a general class of problems arising in exploration geophysics by discovering graphical structure that is physically meaningful and provide evidence of its advantages over alternative approaches.

This paper proposes a model of spatio-temporal dynamics that modelsa global latent process that governs the interactions betweenhigh level clusters of points together with a local observed processin which interactions are decoupled from points outside of one's cluster.Both levels can be thought of as vector autoregressive models.The authors apply their method to modeling data from a numericalsimulation of a geologic model of fluid flow under the earth's subsurface.In general, this was a technically strong paper and shows off some state of the artoptimization techniques.  On the other hand it was also a difficult paper to get through asit was notation heavy and quite dense.  In particular, there are topicsthat are not typically ``NIPS'' topics, for which I think the authors should spendmore time providing intuition for.As an example, I do not have a good graspwhat it means to approximate the latentprocess with a 1-dimensional convection diffusion system --- the authors talk aboutthe computational benefits --- but what kind of behavior can we not capturewhen we make this approximation?The experimental setting seemed interesting but I would have liked to see some morediscussion about the task --- how was the training/test split performed inthis setting?  And how are errors measured? (The authors report errors in percentage,but it is unclear what the denominator here is).I would have also like to see more extensive experiments, particularly with more than one dataset,as it is difficult to understand if the proposed model is generally applicable or if itis specifically good for this fluid-flow setting.  Though spatio-temporal processesthat have interesting things happening at multiple scales are ubiquitous, the paperdoes make assumptions (such as the 1-dimensional convection-diffusion assumption again)that make the specific optimization approach tractable --- but the authors do notdiscuss other settings in which these assumptions might also apply.typos:Next in Section 3, relaxations to simply the problem along with with efficient algorithmfor parameter estimation are developed.unbalanced bracket on line 184page 5 second to last line (268) --- parentheses at wrong level for alpha^{(n)}grep for nonhierarchal and hierarchal -> nonhierarchical, hierarchical


Learning to Optimize via Information-Directed Sampling
We propose information-directed sampling -a new algorithm for online optimization problems in which a decision-maker must balance between exploration and exploitation while learning from partial feedback. Each action is sampled in a manner that minimizes the ratio between the square of expected single-period regret and a measure of information gain: the mutual information between the optimal action and the next observation. We establish an expected regret bound for information-directed sampling that applies across a very general class of models and scales with the entropy of the optimal action distribution. For the widely studied Bernoulli and linear bandit models, we demonstrate simulation performance surpassing popular approaches, including upper confidence bound algorithms, Thompson sampling, and knowledge gradient. Further, we present simple analytic examples illustrating that informationdirected sampling can dramatically outperform upper confidence bound algorithms and Thompson sampling due to the way it measures information gain.

This paper presents a new general algorithm for Bayesian bandit problems, based on a tradeoff between the "information gain" of an action and its contribution to the regret. The idea is new and interesting, and the algorithm appears to display good performance on numerical simulations. However the general results stated for the method are comparable to that of other approaches and the claim that it can "dramatically outperform upper confidence bound algorithms and Thompson sampling" (in the introduction) appear to be excessive in light of the elements provided in Section 5 and 6. In addition, details of the algorithms given in the supplementary material suggest that the implementation is much more involved than that of UCB or Thompson Sampling. Regarding the latter aspect it is difficult to understand by reading the paper to which extend the algorithm can really be implemented and will be efficient. In particular, for which bandit problems are the quantities \alpha_t, g_t, \Delta_t easy to compute? The implementation for Beta-Bernoulli models can only be found in the supplementary material, and the authors almost never mention this classical example of bandit model in the main text. Explanations on how to compute g_t and \Delta_t should be given as early in the text as possible (and certainly not in the supplementary material only).The intuition on why the information gain is defined this way is not obvious: there are plenty of ways of realizing a tradoff between the two quantities \Delta_t and g_t: why not minimizing \Delta_t + 1/g_t for example? The proof of Proposition 2 should be provided in the main text, as (1) it is the theoretical cornerstone of the analysis (the bounds on \Phi_t(\pi^TS) following from previous work if I get it right); (2) it helps understanding why \Pi_t is chosen this way (or at least why this choice make the proof work!); (3) it is short.  The value of the counter-examples in Section 5 is somewhat uncertain as they appear to rely on very specific assumptions. In particular, the first counter-example relates to the very artificial setting of sparse linear bandits without observation noise and known 1-sparsity. In light of the authors' answer in the rebuttal regarding this comment, it is clear that the argument also relies on the fact that the arms are L1-normalized vectors of {0,1}^d, as omitting the normalization would make Thompson sampling select arms with more than one non-zero coordinate (just as the proposed method does). Note that in this (artificial) setting, L1-normalizing the arms is a curious choice as it makes all the arms that have more than one non-zero component inadmissible in the linear bandit model as they are convex combinations of others (ie. they could be removed from start, leaving only the d arms that have a single non-zero component). This being said, the main question here is what can be learned more generally from these very specific situations?In the numerical experiments, please specify that Figure (a) displays the Bayesian regret (drawing problems from the priors). You should also comment more on the difficulty of implementing IDS in the Bernoulli case compared to the other approaches. There is also big typo in the Lai and Robbins lower bound: the sum is before the fraction! (line 394). I didn't check the complexity of implementation for linear bandits: could you comment on how much complex it is compared to Thompson Sampling? Could you also give precisions on the modified version on GP-UCB that is mentioned in the text? In Section 1, it may be worth mentioning that the use of the mutual information to select the most informative actions is also a largely used principle in sensor networks (googling "information-based sensor" of "information-theoretic sensor" will bring up plenty of related references), although the combination with a regret measure is quite probably novel.Minor typos: 124: It's -> It is; 150: $R_t$ should be $R$ I guess; 177: "_the_ design"; 273: it should be \cP_t (the subscript t is missing); 562: 988888 ???.


A Residual Bootstrap for High-Dimensional Regression with Near Low-Rank Designs
We study the residual bootstrap (RB) method in the context of high-dimensional linear regression. Specifically, we analyze the distributional approximation of linear contrasts c ( β ρ − β), where β ρ is a ridge-regression estimator. When regression coefficients are estimated via least squares, classical results show that RB consistently approximates the laws of contrasts, provided that p n, where the design matrix is of size n × p. Up to now, relatively little work has considered how additional structure in the linear model may extend the validity of RB to the setting where p/n 1. In this setting, we propose a version of RB that resamples residuals obtained from ridge regression. Our main structural assumption on the design matrix is that it is nearly low rank -in the sense that its singular values decay according to a power-law profile. Under a few extra technical assumptions, we derive a simple criterion for ensuring that RB consistently approximates the law of a given contrast. We then specialize this result to study confidence intervals for mean response values X i β, where X i is the ith row of the design. More precisely, we show that conditionally on a Gaussian design with near low-rank structure, RB simultaneously approximates all of the laws X i ( β ρ − β), i = 1, . . . , n. This result is also notable as it imposes no sparsity assumptions on β. Furthermore, since our consistency results are formulated in terms of the Mallows (Kantorovich) metric, the existence of a limiting distribution is not required.

The authors propose a theoretical analysis of the residual bootstrap for theridge regression estimator, under general (additive) noise model. The analysis provided requires some structural assumptionthat the empirical covariance of the variables is nearly low rank (in the sensethat its spectrum decreases polynomially fast toward zero). Under some moretechnical assumptions (e.g. the design matrix is Gaussian) the authors prove that the residual bootstrap approximates all the conditional laws of all fitted values.Quality: The paper is very well written and the organization is clear.It gives new insight on the benefit of the residual bootstrap in possibly large dimension without sparsity assumption, even when n \approx p.Clarity: very good.Originality: proof techniques seems rather standard, but the context is new I believe.Significance: this is a good direction to get prediction interval in large dimension.Such a road has still not been carefully explored until recently (apart from few papers,that the authors mentioned). Line by line comments:032: requires imposes ->imposes125: harmonize the way paper are cited (e.g. B&F 1983 is [17]!)131: distribution distribution->distribution133: focus on role-> focus on the role138: Theorem ?? -> provide the correct name (potentially issue with the supplementary version one)157: the bold notation for the conditional sign | if fairly unusual and looks weird.167: \Phi is not defined before, only \Phi_\rho was defined in (4)Summary of the review:175: correct the recurring error: X^\top X+\rho : the first term is a matrix, the second one is a scalar.The identity matrix is missing. This is everywhere in the article / supplementary.289: a variety bounds ->a variety of bounds310/313: can the authors give some examples where such an assumption is realistic in practice?323:that \beta that->that \beta413:a conclusion is missing, no? And some room is available to do so!416: names are duplicatedMost references: please provide the full name when citing the and double check the wrong ones like [6,8,9,10,12] where the "et al." are useless.Note also that if you want to provide first names, do it equally for all authors (sometimes being the same):e.g.: P. J. Bickel and Peter J. Bickel refer to the same authors!441: citation issue again449: idem507/530: the constant could be tracked I guess here. Overall details are a bit loose around here.


Quantized Estimation of Gaussian Sequence Models in Euclidean Balls
A central result in statistical theory is Pinsker's theorem, which characterizes the minimax rate in the normal means model of nonparametric estimation. In this paper, we present an extension to Pinsker's theorem where estimation is carried out under storage or communication constraints. In particular, we place limits on the number of bits used to encode an estimator, and analyze the excess risk in terms of this constraint, the signal size, and the noise level. We give sharp upper and lower bounds for the case of a Euclidean ball, which establishes the Pareto-optimal minimax tradeoff between storage and risk in this setting.

The authors consider Pinsker's theorem on the minimax rate in nonparametric estimation with squared error risk of normal means in an L2 ball. They extend this theorem to a rate-distortion-like setting, where there is a bound on the number of bits used in estimation. For bitrates (nr of bits used per outcome) bounded by B, the authors derive exact formulas for the n -> infinity limit of the minimax risk. As one would expect, the resulting formula becomes Pinsker's formula as B -> infinity, and degenerates to the radius of the L2 ball as B -> 0. QUALITYThe paper seems technically sound (though I did not check all details). While the techniques used are, in the end, variations on standard techniques in rate distortion theory (Theorem 2) and 'establishing-minimax risk lower bounds'-literature (Theorem 1), the specific details are quite different from earlier works and seem pretty nontrivial, in fact quite difficult.   CLARITYThe paper is really quite well-written. Inevitably it requires some prior knowledge on the readers' part but the authors did their best I think. ORIGINALITYTo the best of my knowledge (note though that I am not very familiar with the recent literature) the results presented here are new.Still (and this is my one qualm with the paper), the techniques on which they build are not new at all, and little insight is given in how much of the current work is already 'almost known' or 'rather to be expected' given earlier work. In this respect I think that the third paragraph in Section 5, Discussion, is really far too restricted and, should the paper be accepted, should be much elaborated on - even if this means cutting some other material. In fact, I checked the Yang and Barron (YB) paper quoted by the authors in Section 5 and found that the connections seem to run deeper than suggested. Specifically, the YB proof of the YB Theorem 1 is quite similar to the line of reasoning followed by the authors to prove the author's Theorem 1. Let me elaborate on this: while in the classical minimax risk lower bounds as established by YB, there are indeed no constraints on the nr of bits used to store/communicate the estimator (so in this sense the authors are correct that they 'pursue a different direction'), YB's *proof* of such unconstrained bounds does proceed by first looking at restricted set of potential 'truths' (one puts an \epsilon-net on the set \Theta_n) and putting a discrete prior w on them, and giving (via Fano's inequality) a lower bound on the risk of the estimators which are optimal relative to w. Such estimators will *automatically* be representable in a restricted nr of bits (log of the size of the \epsilon-net, to be precise), and thus automatically satisfy a constraint of the type the authors impose. This leads one to wonder whether there isn't already 'implicit rate distortion' going on in classical minimax risk lower bounds, and whether the results in the present paper cannot be more easily re-derived as simple corollaries of existing results. I found it strange that none of this is discussed in the paper. So all in all I would say that the *details* of the paper are certainly original - the result is new, the precise proofs are new and highly nontrivial, but the 'general type of result' is less new than the authors suggest; this is not a reason for rejection per se but it definitely needs more attention. SIGNIFICANCEFor the same reasons I gave as to the originality, I think that while there is reasonable significance, it is not perhaps as high as the paper suggests.PRO/CON-AnalysisPRO: well-written, highly nontrivial and useful result.CONTRA: builds heavily on earlier work, and precise relation to it is not discussed very wellCOMMENT ON IMPACT SCORE: I had a hard time choosing, I'm not really happy there are just two options here. In the end I chose 'incremental' because, while the setting is different and quite novel (marrying rate distortion and minimax nonparametric analysis), the way this is analyzed is *very* similar to existing methods in both fields ; and that these methods are interrelated was already quite well-known. So, mathematically speaking, the term 'incremental' seems justified.


Delay-Tolerant Algorithms for Asynchronous Distributed Online Learning
We analyze new online gradient descent algorithms for distributed systems with large delays between gradient computations and the corresponding updates. Using insights from adaptive gradient methods, we develop algorithms that adapt not only to the sequence of gradients, but also to the precise update delays that occur. We first give an impractical algorithm that achieves a regret bound that precisely quantifies the impact of the delays. We then analyze AdaptiveRevision, an algorithm that is efficiently implementable and achieves comparable guarantees. The key algorithmic technique is appropriately and efficiently revising the learning rate used for previous gradient steps. Experimental results show when the delays grow large (1000 updates or more), our new algorithms perform significantly better than standard adaptive gradient methods.

The authors proposed a delay-tolerant algorithm based on asynchronous distributed online learning method.  The authors reordered the summation operation in the optimization process and achieved a more efficient asynchronous distributed optimization.  the authors also proved a bound of the regret with the unprojected online gradient descent algorithm, while the bound seems not very tight.  experiments shows the proposed algorithm performs much better than other async algorithm with many threads. There are two concerns with the results.One concern is the authors did not report the time comparison.  As we know the main contribution of the paper is to replace the sync with asynchronous updating in the distributed environment.  The overhead of sync is critical and major limitation of the distributed optimization. The authors proposed new algorithms which use async and achieve comparable performance with the sync algorithm which looks very promising, while the lack of time comparison, I can not tell how much improvement or contribution the algorithm can gain over existing algorithms.  So the contribution of the paper is heavily discounted without this report.Another concern is the authors seems to update each dimension separately, this is valuable in handling high dimensional data.  While it does not state that the algorithm can also handle the long matrix optimization.  In general the high dimensional problem is very rare, most of the cases we need to handle the big samples.  Does the algorithm have any advantage over existing methods when dimension is low while sample size is huge?  Also I suggest the authors perform more experiments to report the results in two cases.


Discriminative Metric Learning by Neighborhood Gerrymandering
We formulate the problem of metric learning for k nearest neighbor classification as a large margin structured prediction problem, with a latent variable representing the choice of neighbors and the task loss directly corresponding to classification error. We describe an efficient algorithm for exact loss augmented inference, and a fast gradient descent algorithm for learning in this model. The objective drives the metric to establish neighborhood boundaries that benefit the true class labels for the training points. Our approach, reminiscent of gerrymandering (redrawing of political boundaries to provide advantage to certain parties), is more direct in its handling of optimizing classification accuracy than those previously proposed. In experiments on a variety of data sets our method is shown to achieve excellent results compared to current state of the art in metric learning.

The paper proposes a new approach to Mahalanobis metric learning which views metric learning as a structure-prediction problem. The structure we seek to establish is the set, $h$, of appropriate k-Nearest Neighbors (k-NNs) of an instance; appropriate in the sense that it will lead to a small classification error.  The idea of the authors is quite elegant: in learning a metric for k-NN prediction we can have a correct prediction for an instance $x_i$ as long as the majority of its k-NNs are of the same class as $x_i$. The metric learning algorithm they propose does exactly that, i.e. it looks for the Mahalanobis metric matrix $W$ that leads to k-nearest neighborhoods for each $x_i$ instance such that the majority of the instances of the neighborhood belongs to the correct class, i.e. that of $x_i$. Note here that there can be many such neighborhoods that predict the correct label for a given instance $x_i$, it is enough to prefer at least one of them over all the ones that predict the wrong label.  The authors define a structure prediction problem in which they optimize a tractable surrogate loss which is parametrized as a Mahalanobis distance in which for each instance the structure sought is the set of instances $h$ k-NNs that will have zero classification error. The authors build directly on the work of [12], McFee and Lankriet "Metric Learning to rank", ICML 2010, where metrics were learned to predict another type of structure, related to the k-NN, rankings.  They use the same feature map $\Psi(x,h)$ to quantify the relation of an instance $x$ to the $h$ structure (which is defined as the matrix that contains the negatives of the sums of the squared pairwise attribute differences of $x$ and its neighbors in $h$). This map under the Frobenious inner product with a PSD matrix $W$ leads to the sum of the negative Mahalanobis distances of $x$ from the instances of $h$, measuring in a sense the similarity of $x$ to $h$, S_W(x,h). The set of instances that maximizes S_W is the set of k-NNs of $x$, and the standard classification loss is the difference between the true label $x$ and the majority label of $h. Since this is intractable the authors define a tractable loss on the basis of the neighborhood that mostly violates the correct classication, using the sum of its loss (which plays the role of the margin) and its similarity to $x$, and the neighborhood that results in a correct classification and has the largest similarity to $x$.  The minimize the loss together with a frobenius norm regularizer on $W$. The minimization is done by stochastic gradient descent at every step of which they solve two neighborhood inference problems. One inference problem finds the neighborhood that has maximum similarity and classiciation-loss (loss augmented inference) and the other finds the maximum similarity zero classification-loss neighborhood.  The gradient update of the loss is then computed over these two inferenced values. The update rule learns a W such that it pulls instances close together if they belong in the zero classification loss neighborhood or far apart if the belong in a neighborhood that mostly violates the correct classication constraints, all other instances do not influence the updates. At each step there is a projection on the PSD matrix cone. In the experimental section the authors compare their method against LMNN, MLR (the metric learning to rank method), and the gradient boosted variant of LMNN on six datasets with small to medium numbers of instances, up to 20k and a few hundreds of features. The evaluated the performance of the different methods under k-NN, with k-NN=3,7,11. The empirical results show an advantage of their method over the baselines.  Overall I quite liked the idea presented in the paper and the definition of metric learning in which one has to find the right neighborhood, formulating the problem as a structure prediction problem. Even though it builds considerably on [12] I think that the optimization over the neighborhoods makes it substantially different from that paper, as well as interesting. One of the critisisms of the prior work and in particular LMNN is the fact that the target neighborhood structure remains fixed over the optimization. There are extensions of LMNN that do the update of the neighborhood as a part of learning, lifting that constraint, e.g. Jun Wang et al, Learning Neighborhoods for Metric Learning, ECML/PKDD 2012; the authors should discuss them and ideally include them as a baseline in the experiments since they also learn the neighborhood.  I am missing a discussion on the computational complexity especically in view of the small to medium size of the datasets used in the evaluation. How fast does the method converge, what are the problem sizes to which it scales. Also the experimental section is a bit weak, the hyper-parameters of the different methods as well as the grids from which they take values are not described fully. In addition I would have welcomed a more extensive methods with other metric methods which have been shown to have good predictive performance, for example Jun Wang et al, Parametric Local Metric Learning for Nearest Neighbor Classification, in NIPS 2012, which has been shown to have better performance than the baseline methods against which the authors compare.