Automatic Variational Inference in Stan
Variational inference is a scalable technique for approximate Bayesian inference. Deriving variational inference algorithms requires tedious model-specific calculations; this makes it difficult for non-experts to use. We propose an automatic variational inference algorithm, automatic differentiation variational inference ( ); we implement it in Stan (code available), a probabilistic programming system. In the user provides a Bayesian model and a dataset, nothing else. We make no conjugacy assumptions and support a broad class of models. The algorithm automatically determines an appropriate variational family and optimizes the variational objective. We compare to sampling across hierarchical generalized linear models, nonconjugate matrix factorization, and a mixture model. We train the mixture model on a quarter million images. With we can use variational inference on any model we write in Stan.

Review Summary ============== Score: 8 top 50% of NIPS papersI'm excited about the core ideas in this paper: it's a huge step towards making variational inference general-purpose at scale for a big class of models. I think it belongs at NIPS, especially if the authors can address some concerns about local optima and revise the paper to include a new plot with a more intuitive illustrative example.Summary of Paper ================ This paper develops a method for automatic variational inference in the Stan probabilistic programming language, which can support any model that has a differentiable log joint probability. The proposed method works by transforming all latent variables to unconstrained real values, and then approximating their posterior with a Gaussian with diagonal covariance (the diagonal assumption corresponds to an mean-field independence assumption on every scalar in the parameter vector). A further transformation standardizes each 1D Gaussian to a standard normal (zero mean, unit variance), so that Monte Carlo integration can be used to estimate the expectations involved in the variational objective function. To scale to massive datasets, the authors employ stochastic variational inference with an adaptive learning rate. Experiments show that the method provides better predictions on heldout data than Stan's MCMC algorithm for a variety of nonconjugate models (regression, matrix factorization, and mixtures). Reviewer Comments on Method =========================== Overall, I like the general approach used. I have a few key issues I'd like addressed in the rebuttal, in order of importance.Issue M1: I'd like to see at least some discussion of how vulnerable this method is to local optima. My guess is, that like other variational methods, the answer is extremely sensitive. It would be interesting to hear if the authors have any strategies to avoid this sensitivity, aside from multiple random initializations.Issue M2: I'm concerned that the monte-carlo integration strategy may be very noisy and lead to slow convergence. Previous work on black-box variational (Ranganath, Gerish and Blei 2014) indicated that without some strategies to reduce variance (rao-blackwell-ization and control variate methods), simple monte carlo estimates to obtain gradients were quite terrible -- see esp. their Fig. 2. I'd like to know if this kind of diagnostic plot, or any other, was ever made. The experiments seem to converge OK, but I find it really surprising that a single sample (M=1), as used in all experiments, is sufficient.Issue M3: I'd like to see more discussion of how the method is initialized. This is a crucial for variational inference methods that get stuck easily in local optima. Have different strategies been tried? Issue M4: How to select batch size? It seems like the growing consensus is that very small batches will lead to poor models, which makes sense. But is there any advice to practioners about how to select an appropriate batch size? Perhaps just use the largest size you can afford, doing full-dataset inference if feasible? Reviewer Comments on Experiments ================================ Issue E1: Internal experiments should show many (at least 5, preferablly more like 20 or 50) runs of each method (each from an independent initialization), rather than just one run. Local optima are very prevalent in inference, and plots of multiple runs can help diagnose how sensitive each method (MCMC/ADVI) is.Issue E2: Experiments are missing one "intuition-building" plot, that shows the estimated parameters in a case for which the good or ideal parameters are "obvious" from inspection of the raw data. I like the faces in Fig 5c, but its hard to tell if this is a "good" result. Instead, I recommend something like a classic mixture model on 2D scatterplot data with a few obvious clusters, some of which have non-trivial covariance. The result from ADVI should be some Gaussian blobs centered on cluster means but with diagonal covariances (due to mean-field), while the MCMC result maybe be more accurate (no independence assumptions) but probably slower. I think this will help readers understand the tradeoffs of mean-field, etc. It's often really hard to understand from traceplots whether a given run is "good" in an absolute sense.Issue E3: In Line 414, "label switching" is blamed for the poor performance of MCMC for the mixture model. I don't think I buy this explanation. The fact that cluster indices can be permutted over many iterations doesn't explain poor predictive performance, because prediction scores should be irrelevant to the order of the clusters. Instead, the problem is probably that the MCMC sampler is stuck in a local optima and cannot escape. This is the kind of thing that is best diagnosed with the plot from E2. Less Critical Issues (need not reply to in rebuttal) ---------------------------------------------------- Issue M5: Variational and MCMC are often seen as either/or choices, but I'm often interested in whether variational could be used to rapidly get to a decent estimate, and then MCMC is used to get a more refined, reliable estimate. Is it possible to initialize an MCMC chain in Stan with the output of variational? Or vice versa?Issue E4: This may be too much to ask, but I think it would be nice to include one comparison to an external method (variational probably, but MCMC could work too) that has been custom-designed for some specific data and model. For many readers, the question of whether they should use Stan or roll-their-own custom solution may be relevant, and it would be interesting to show a case for which the Stan approach reaches comparable performance without the cost of custom inference. Significance ============ This project has great potential for wide-adoption, given its complete integration into the increasingly-popular Stan language. Novelty ======= Developing the proposed general-purpose framework and integrating it into Stan is definitely a very new and worthwhile contribution.  Clarity ======= The paper has a crisp, simple style that I found very accessible. Aside from a few unclear sentences (detailed below), I think the overall organization and style do not need heavy revision. Line-by-line Comments ===================== (not necessary to respond to these in rebuttal)Line 257: I found the phrase "we cannot directly represent it as a computer program" a bit confusing. Perhaps just leave it out. The sentence about an intractable integral is enough.Supplement E, Line 589: You give ref 3 (the SVI JMLR article) as a reference for adagrad, but in fact that paper has very little material on adaptive gradients, aside from a reference other papers. Please fix.


Estimating Jaccard Index with Missing Observations: A Matrix Calibration Approach
The Jaccard index is a standard statistics for comparing the pairwise similarity between data samples. This paper investigates the problem of estimating a Jaccard index matrix when there are missing observations in data samples. Starting from a Jaccard index matrix approximated from the incomplete data, our method calibrates the matrix to meet the requirement of positive semi-definiteness and other constraints, through a simple alternating projection algorithm. Compared with conventional approaches that estimate the similarity matrix based on the imputed data, our method has a strong advantage in that the calibrated matrix is guaranteed to be closer to the unknown ground truth in the Frobenius norm than the un-calibrated matrix (except in special cases they are identical). We carried out a series of empirical experiments and the results confirmed our theoretical justification. The evaluation also reported significantly improved results in real learning tasks on benchmark datasets.

The paper was clear and concise. The idea is original as far as the reviewer is aware, and after a modest amount of literature search. The experimental results are convincing enough to warrant considering the approach for small to medium sized problems, with sample sizes in the order of 1K-10K, and the ratio of observed features is in the .2-.5 range. It's not clear from the experimental evaluation, however, whether the approach is as effective, compared to the other imputation approaches, on larger problems (larger similarity matrices) when projecting onto the set of positive semi-definite matrices - which relies on computation of SVD - becomes intractable or requires approximate methods (the authors do not clarify whether approximate SVD was used in the experiments). As the authors point out, the matrix decomposition is a major computational bottleneck and limitation of this method, and as such the reviewer would argue that this would make it an unappealing method to apply to many real world problems where sample sizes are much larger that 10K. This (what happens with larger data sets) is a major concern. Another concern is the relevance of this whole apparatus. I am not aware, off the cuff, of a practical situation in which one would want to compute Jaccard similarity matrix while facing missing values in the samples. It is quite possible that such a situation comes up, or even is very important, but the authors fail to provide any practical examples. All of the experiments are on data sets (like MNIST or USPS) in which the values are dropped artificially. Obviously, such experiments are crucial to evaluate the accuracy of the proposed method under controlled circumstances, and to measure its sensitivity to various parameters. But equally crucial, I think, is to provide at least one example in which real data, with real underlying phenomenon of interest (here the missing values) is examined. Since in real data missing values are most likely not distributed uniformly (see below more on this) it would be potentially a different "game" and the outcome, in particular the relative standing of the proposed method vs. imputation, is uncertain. I'd welcome report on any additional experiment along these lines that the authors can include in the rebuttal.Another suggestion for improvement, related to the comment above: consider cases other than uniformly random missing features. At some point in continuum from high to low entropy distributions one could expect imputation via kNN to outperform.


Action-Conditional Video Prediction using Deep Networks in Atari Games
Motivated by vision-based reinforcement learning (RL) problems, in particular Atari games from the recent benchmark Aracade Learning Environment (ALE), we consider spatio-temporal prediction problems where future image-frames depend on control variables or actions as well as previous frames. While not composed of natural scenes, frames in Atari games are high-dimensional in size, can involve tens of objects with one or more objects being controlled by the actions directly and many other objects being influenced indirectly, can involve entry and departure of objects, and can involve deep partial observability. We propose and evaluate two deep neural network architectures that consist of encoding, actionconditional transformation, and decoding layers based on convolutional neural networks and recurrent neural networks. Experimental results show that the proposed architectures are able to generate visually-realistic frames that are also useful for control over approximately 100-step action-conditional futures in some games. To the best of our knowledge, this paper is the first to make and evaluate long-term predictions on high-dimensional video conditioned by control inputs.

The authors present a system for video frame prediction in ATARI games taking into account the game actions. The main novelty is a multiplicative transformation layer that selects weights based on a given action vector. To keep the system scalable, a factorization of the weight tensor is used. Encoding and decoding of images is done by a well-established convolutional network architecture.They propose two different architectures: a feed-forward network that gets a fixed number of previous frames as input and a recurrent network that gets only one frame but has an LSTM layer before the transformation layer. (I think at least once in the paper they should say what's an LSTM and what the abbreviation stands for.)The qualitative assessment of the generated frames as shown by the supplementary videos is convincing. A quantitative evaluation of mean squared pixel error is given in a clear way but it is not surprising that their system beats a linear and a nonlinear predictor that don't take into account the game actions.Another experiment in which the generated frames are used instead of the real frames as input for a DQN agent also shows results as expected. The performance is worse than for real input frames, better that random play, and their system beats the no-action predictors.An interesting application of this system is to use it during training of an agent to improve exploration. They show that if a DQN agent takes an action that leads to a predicted frame that has least similarity with previously seen frames (rather than a random action), the final performance is significantly improved on some games. Finally it is shown that the system can be used to automatically analyze some of the game dynamics. Game actions that have similar effects can be identified from similarities in the transformation weight matrix. Furthermore the system estimates which pixels of the image are controlled directly by actions and which pixels are uncontrollable game dynamics.The authors write that "To the best of our knowledge, this paper is the first to make and evaluate long-term predictions on high-dimensional video conditioned by control inputs." This may be true, because to my knowledge previous such systems did not have LSTM. I know, however, that there is an old paper by Schmidhuber and Huber from 1991 (International Journal of Neural Systems) where a similar neural predictor learns to predict the next visual input frame of a fovea, given previous input and action. The predictor is embedded in a reinforcement learning system that learns sequences of saccades that lead to desired visual objects defined in a separate goal input. Schmidhuber also had papers at IJCNN 1990 and NIPS 1991 where both the predictor and the action generator were recurrent (but no LSTM yet). Nevertheless, it would be good if the authors could point out what's different in their system, which for example also uses a different RL method.The results are clear, the paper is well structured and suggested for acceptance after minor revisions as indicated above.


Bidirectional Recurrent Neural Networks as Generative Models
Bidirectional recurrent neural networks (RNN) are trained to predict both in the positive and negative time directions simultaneously. They have not been used commonly in unsupervised tasks, because a probabilistic interpretation of the model has been difficult. Recently, two different frameworks, GSN and NADE, provide a connection between reconstruction and probabilistic modeling, which makes the interpretation possible. As far as we know, neither GSN or NADE have been studied in the context of time series before. As an example of an unsupervised task, we study the problem of filling in gaps in high-dimensional time series with complex dynamics. Although unidirectional RNNs have recently been trained successfully to model such time series, inference in the negative time direction is non-trivial. We propose two probabilistic interpretations of bidirectional RNNs that can be used to reconstruct missing gaps efficiently. Our experiments on text data show that both proposed methods are much more accurate than unidirectional reconstructions, although a bit less accurate than a computationally complex bidirectional Bayesian inference on the unidirectional RNN. We also provide results on music data for which the Bayesian inference is computationally infeasible, demonstrating the scalability of the proposed methods.

Summary:The paper proposes a modification of the traditional bidirectional RNN, which predicts output at time t from forward hidden state t-1 and backward hidden state t+1, without getting information from x_t. This allows training the network to map directly from inputs X to outputs Y. The authors test the capability of this architecture to compute the likelihood of completed gaps in sequences, using GSN and NADE - the results are compared to "Bayesian MCMC" inference in standard RNNs. The paper shows that for longer gaps, GSN using the specified BRNN architecture is computationally cheaper and competitive in quality with RNNs and NADE is inferior in quality to GSN. Quality and Clarity The models and the inference, which was used to compute the NLL scores, are both clearly described. I did not see major technical or experimental setup issues.- The "one-way inference" baseline seems to outperform the BRNN for the first couple of characters (Fig 2, left side). Why is it missing in the right plot?- Given the good performance of "one-way inference" on the first couple of characters (Fig 2), wouldn't it make sense to also compare to the setup where we have 2 RNNs, one trained to predict characters going forward, one trained to predict going backward, and then we pick the first two characters from the forward RNN, last two characters from the backward RNN and perhaps the middle by voting? How would this baseline have done? This naive approach may well break down for longer sequences, but if it dominates it would suggest the experimental setup being "too easy" -- a gap of 5 characters may not be sufficient to show significant BRNN benefits.- Why is GSN better than NADE on wikipedia in Fig 1 yet appearing noticeably worse for wikipedia in Fig 2?- For Bayesian MCMC on RNNs (eq 10), when the gap is large making it hard to explore every single variable setting, is it possible to approximately estimate the posterior using direct MCMC sampling in the RNN rather than explicitly enumerating all possible missing states?Originality The modification of BRNN that connects each output y_t to forward hidden state x_{t-1} and to backward hidden state x_{t+1} appears novel to the best of my knowledge. It is a rather straightforward extension of the standard RNN. I am also not aware of existing experiments on evaluating the performance of such a model for likelihood estimation of sequence gaps task, although the integration of the BRNN with GSN and NADE is rather straightforward. Significance I am not convinced that the task of estimating the likelihood of sequence completions has much practical interest. While it is a convenient way to evaluate model quality, getting MAP estimates (e.g. using beam search) may be sufficient for most practical uses?While the BRNN is faster than the Bayesian MCMC, the quality seems a bit inferior. I am curious if there is a way to achieve getting the estimates using MCMC and RNNs without enumerating all states? That would be a reasonable alternative.


Variance Reduced Stochastic Gradient Descent with Neighbors
Stochastic Gradient Descent (SGD) is a workhorse in machine learning, yet its slow convergence can be a computational bottleneck. Variance reduction techniques such as SAG, SVRG and SAGA have been proposed to overcome this weakness, achieving linear convergence. However, these methods are either based on computations of full gradients at pivot points, or on keeping per data point corrections in memory. Therefore speed-ups relative to SGD may need a minimal number of epochs in order to materialize. This paper investigates algorithms that can exploit neighborhood structure in the training data to share and re-use information about past stochastic gradients across data points, which offers advantages in the transient optimization phase. As a side-product we provide a unified convergence analysis for a family of variance reduction algorithms, which we call memorization algorithms. We provide experimental results supporting our theory.

-- Comments after rebuttal -- The rebuttal from the authors was concise.However, I was not convinced about the assumption in Eq. 14 of the paper and how theauthors defended it. The authors say: "Many documents (text categorization),[..] or time series signals (speech recognition) in a training set are alike. This fact is not systematically exploited by any existing stochastic optimization method!" I don't think this is correct. In fact I think this is what statistical learning is all about,i.e., exploit similarities in data. Also in their point (B.2) the reviewers say "Eq. 14 should not be seen as an assumption but rather as one part of a trade-off. We can make \eta arbitrarily small by restricting the size of the neighborhood." These trade-offs are very hard to do in practice. This is in fact where the crux of their method lies. It would be very interesting if the authors had a concrete answer to that.Still, I slightly increased my rating for this paper because, judging by the rebuttal,it seems that the authors understand the shortcomings of this paper and how to resolve them. This should allow the paper to get published if other reviewers are convinced by the rebuttal. ---- The authors study a general formulation of stochastic variance reduction (SVRG)methods and derive a non-asymptotic analysis under strong convexity.This work is ambitious and creative. The idea of using neighborhoodstructure between data points to share information is very smart,and looks natural only in hindsight.However, by reading this paper I felt that it is work under progress with severalimportant details left unfinished. If I am mistaken I would be happy to recommend an accept.On a high-level the paper aims to do many things at the same time, e.g.,have a common analytic framework for SVRG methods, introduce and analyze theidea of data neighborhoods in stochastic approximations, apply SVRG in streaming settings, etc. As a reader I felt that none of these goals were fully achieved.   Regarding the common analytic framework of SVRG methodsI struggled with assumptions in Ineq. (14) and the uniformity assumptions for the memorizationalgorithms (e.g., Lines #213-215).Why are these assumptions reasonable? I am especially skeptical of Ineq. (14) because it uniformly bounds all past and present gradients and basically assumes everything is fine with the stochastic approximation. Although I could buy such an assumption for implicit procedures (e.g.,implicit SGD) which often have these uniform bounds,explicit procedures as in Eq. (2) generally do not.Regarding the streaming setting, I am confused how the authors define the streamingsetting and how they analyze and test in practice. In Line #411 the authors note: "We have found very similar results in a streaming setting, i.e., presenting data once in randomorder."I don't think this defines a streaming setting, and also seems to disagree with authors'discussion on the streaming setting in Abstract and intro sections. My own understanding is that in streaming settings one has access to infinitedata streamed one-by-one and then discarded. One key difficulty in the analysis of SVRG procedures in streaming settings is that one has no access to unbiased estimates of the full gradient.How do the authors cope with this difficulty? I suspect they use Lipschitz bounds as Ineq. (14)however this is not made explicit.Another issue here -- potentially more important -- is thatin streaming settings the estimation problem is different. In finite data sets SVRGmethods -- as well as other methods -- compute the empirical risk minimizer, e.g.,the MLE. However, in streaming settings one estimates the true model parameters. In the former case one approximates an estimator (e.g., MLE) and in the latter case one approximates the true model parameters. Thus, in the streaming case the "linear convergence rate" ofaveraging methods (mentioned by the authors in the abstract) cannot be surpassed. In the non-streaming case one can geometrically converge to the empirical minimizer, howeverthis minimizer is an estimator that is roughly O(1/n) away from the true model parameters. It is the nature of stochastic learning problems where the 1/n simply cannot be surpassed. This is a key issue that causes confusion in stochastic optimization from my own experience. Therefore, it would be important that the authors clarify the streaming setting theymention in the abstract, where it appears in the analysis, and how it is being tested in experiments. Under this light, it is unclear what they mean in Line #411.Finally, the promising idea of using neighbors of data remains underdeveloped. In Lines #138-144 the authors attempt to give practical advice on the matter but they are not specific. I am unsure how one canefficiently define these potentially O(n^2)neighborhood relations. The related analysis is solid using the Lipschitz conditions,but in the experiments it was unclear how the neighborhood SVRG variants were implemented.Other issues: - Figure 3 is not explained in main text. - Captions in figures are not informative. - Line #53, the Robbins-Monro paper is not about SGD. It is about the much more general idea ofstochastic approximations in the streaming setting.


Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach
In this paper we address the problem of decision making within a Markov decision process (MDP) framework where risk and modeling errors are taken into account. Our approach is to minimize a risk-sensitive conditional-value-at-risk (CVaR) objective, as opposed to a standard risk-neutral expectation. We refer to such problem as CVaR MDP. Our first contribution is to show that a CVaR objective, besides capturing risk sensitivity, has an alternative interpretation as expected cost under worst-case modeling errors, for a given error budget. This result, which is of independent interest, motivates CVaR MDPs as a unifying framework for risk-sensitive and robust decision making. Our second contribution is to present an approximate value-iteration algorithm for CVaR MDPs and analyze its convergence rate. To our knowledge, this is the first solution algorithm for CVaR MDPs that enjoys error guarantees. Finally, we present results from numerical experiments that corroborate our theoretical findings and show the practicality of our approach.

The paper shows that a CVaR objective has an alternative interpretation as expected cost under worst-case modeling errors, if modeling errors are bounded by a budjet, bounding their product.The paper also introduces an approximate value iteration algorithm for CVaR MDPs, which is to the best of their knowledge the first solution to CVaR MDPs with convergence error guarantees, and the first algorithm to approximate globally optimal policies for CVaR MDPs. Points that require improvement are:  (1) While proofs can be found in the supplementary material, it   seems necessary to have at least proof sketches inside the paper,   especially given that one of the two main contributions of the paper   is the proof of proposition 1. (2) The paper heavily relies on an error budget constraint that     seems non-intuitive/non-realistic to me. It would be good to     explain whether this error-budget just makes the math easier, or     does it also have a realistic interpretation. (3) Experiment with a more complex domain could strenghten the pThe paper is of high-quality, and very well-presented.The paper provides significant insight on the connection between risk-sensitivity and robustness, as well as a practical algorithm for discrete state/action CVAR MDPs. Robust MDPs are an important research problem, and the paper makes progress towards better understanding and solving them.Detailed comments by order of appearance: ========================================= - line 39 'worst-case scenario' - what does it mean: worst-case cost under worst-case parameters? or expected-cost under worst-case parameters?- line 47 - why purturbing only transition probabilities, and not reward as well? would your result hold with reward purturbation? Some clarification is needed here, and possibly in the abstract.- line 79 - 'compute globally optimal' - since your algorithm is an approximate VI, wouldn't it be more accurate to say "approximate globally optimal..." and mention the dependence on interpolation resolution?- line 119 'our results easily generalize' - if they indeed generalize, it would be good to briefly explain how at a later section of the paper.- line 122 it should be H x A x X - line 128 you use Z_t for C(x,a) - please briefly clarify the connection explicitely- line 144 'worst-case expected ... in presence of worst-case' => 'expected ... in presence of worst-case' Also, could you relate 'expected reward under worst-case purturbation'to 'worst-case reward under worst-case purturbation' and possibly toto 'worst-case reward under expected purturbation'?- line 148 - (x_0...x_T) - it's not clear if you include actions in the trajectory, and if not, why not (later it becomes clearer when you use a deterministic, fixed policy)- line 151 - why (x_0...x_T) starting from x_1, especially given that cost is C(x,a)?- line 165 - the error budget seems uninutitive to me. Beyond the fact that it's not clear why errors should be bounded by a budget, this specific form of budget which bounds the product of errors seem uninutitive.Other than helping the math, does this come in realistic contexts? if yes, could you demonstrate?Also does your algorithm needs to know the budget (\eta) in advance? if not, what happens when: the assumed budget is smaller than the actual errors? if yes, is there a way to find a budget (upper-bound) efficiently in the presence of the exponential number of possible error combinations?Also, it's not clear to my why bounding the delta product from above bounds the errors: very small deltas (e.g. delta=0) applied to 'good' transitions may be as bad - don't they? line 166 - "states that the worst cannot happen at each time" - it sounds more accurate to say "*can* state that the worst...", since if the budget is large enough, the worst can happen at each time- While proofs can be found in the supplementary material, it seems necessary to have at least proof sketches inside the paper, especially for proposition 1, andespecially given that some proofs are part of the main claimed contributions of the paper. - line 177 - if you claim the comparison is 'instructive', it seems appropriate, to guide the reader and provide a brief explanation to what is instructive in the comparison.- Theorem 2: it might add to clarify if 'Z' is replaced by something line \mathcal{Z}_t- experiments:  - The experiments verify that your algorithm behaves as expected with   respect to risk and model errors on a grid world example. This is   good as a first step, but it would be better verify your algorithm   on a more complex domain.  - it seems that you built the obstacles in a way that the length of   the optimal path for any given (increasing) safety level is   increasing. It would be good explicitely clarify that (I didn't find   a mention of that in the paper).  - initially it confuses that the obstacles are yellow, given that the   value function is also yellow in parts - it seems like obstacles have   high value  - shouldn't M be negative (since your plotted value function has   negative values)


Sampling from Probabilistic Submodular Models
Submodular and supermodular functions have found wide applicability in machine learning, capturing notions such as diversity and regularity, respectively. These notions have deep consequences for optimization, and the problem of (approximately) optimizing submodular functions has received much attention. However, beyond optimization, these notions allow specifying expressive probabilistic models that can be used to quantify predictive uncertainty via marginal inference. Prominent, well-studied special cases include Ising models and determinantal point processes, but the general class of log-submodular and log-supermodular models is much richer and little studied. In this paper, we investigate the use of Markov chain Monte Carlo sampling to perform approximate inference in general log-submodular and log-supermodular models. In particular, we consider a simple Gibbs sampling procedure, and establish two sufficient conditions, the first guaranteeing polynomial-time, and the second fast (O(n log n)) mixing. We also evaluate the efficiency of the Gibbs sampler on three examples of such models, and compare against a recently proposed variational approach.

The authors provide a formal analysis of the mixing time of simple Gibbs sampling schemes on probabilistic submodular models. The main result is that the mixing time of Gibbs is polynomial or even less nlog(n) with the dimensionality on submodular models. To my knowledge, this work is novel and the result is interesting.Quality: The writing of this paper is clear and most of the paper is easy to follow. However, the author gives little introductions on some key results, like Therom 2, so it is not well self-contained. This makes it hard to understand for people not familiar the specific literature.Strength: The bound of mixing time is important for MCMC methods, but often difficult to establish. This work provides a novel result of the mixing time using submodularity.Questions:In addition, the proof of two main theorems on the bounds is not clear to me. More specific: Theorem 1: The idea of the set of canonical path is clear, but the definition C_{s,s'} seems simply the product space of two configurations, rather than a canonical path between them. This makes it difficult to understand the injective map eta_{s,s'}.Theorem 3: It is not clear to me what is the relationship between the chain (X_t, Y_t) and set (R, S). Is (R,S) the current sample from the pair of chains? In particular, what does it mean the two chain are identical if v = r?One more point is that it is good have more discussion on how realistic the strong condition for fast mixing. It seems very strong to have gamma_{f, beta} < 1 on most interesting models.I think the author should be careful about not over advertising the polynomial mixing time, because the bound is actually exponential to zeta, the notation of distance to modular function. It can be very large in many models, so the bounds may not be a good guid of real mixing time in practice.For the experiment, it is good to compare Gibbs with variational method, but it is not clear to me to see the bound of mixing time in the result.


Differentially Private Subspace Clustering
Subspace clustering is an unsupervised learning problem that aims at grouping data points into multiple "clusters" so that data points in a single cluster lie approximately on a low-dimensional linear subspace. It is originally motivated by 3D motion segmentation in computer vision, but has recently been generically applied to a wide range of statistical machine learning problems, which often involves sensitive datasets about human subjects. This raises a dire concern for data privacy. In this work, we build on the framework of differential privacy and present two provably private subspace clustering algorithms. We demonstrate via both theory and experiments that one of the presented methods enjoys formal privacy and utility guarantees; the other one asymptotically preserves differential privacy while having good performance in practice. Along the course of the proof, we also obtain two new provable guarantees for the agnostic subspace clustering and the graph connectivity problem which might be of independent interests.

General comments:The paper is well written. I would just remove "cryptographic" in the abstract. Not a lot of people are familiar with differential privacy and many of them would get confused with the notion of cryptography in which there exists an algorithm that reverts the "encrypted" data to its original form (I consider that once the data is noisified, its original form is essentially "lost"). Technical comments:Following the argument in the supplementary information of ref. [30], the authors argue that being able to find a perfect SSC clustering breaches the notion of differential privacy (DP). Agreed. * But * the point is that finding a perfect SSC clustering is just computationally impossible.Consider the following problem:Name: SSC approximation. Instance: a set of vectors in R ^ d, integer q\ geq 1, integer k\ geq 1, bound b\ geq 0. Question: does there exist a set of k subspaces of dimension q each such that the subspace clustering objective (as defined by (3) in the paper) is not more than b.This problem is APX-Complete from reduction from Vertex cover (vertices X, edges E): d is the number of vertices, and for each element of E, create one vector with the indicator variable for the two vertices of X. Fix k = the same as in X3C, b = 1, q = 1. Since SSC solution gives orthonormal bases, solution to SSC approximation gives exactly the indexes of the vertices in X which cover all edges (since b = 1).Hence, there exists a constant\ gamma\geq 0 such that if P\ neq NP, SSC approximation is hard to approximate within 1+ \ gamma.So any SSC clustering can only guarantee to find an approximation to the optimum. Finally, SSC clustering is no "tougher" for privacy than other models, like the (more) popular barycenter-based clustering algorithms of k-means/EM.However, of course, it is crucial to address SSC clustering in a private framework.Then, the authors justify the use of a local-sensitivity approach by the fact that global sensitivity would be too high and thus preclude de facto any efficient DP method for the clustering problem. Agreed. * But * then the weaker model somehow imposes to find very efficient algorithms, as otherwise there would be no reason to weaken privacy for (still) inefficient algorithms. From this standpoint, Nissim's approach (ref. [23]) makes clear the fact that the dependence on noise is crucial, and non-trivial to solve even in the SAF framework. In the longer version of their paper, Nissim et al. make this clear (Section 5.1 of www.cse.psu.edu / ~ads22 / pubs / NRS07 / NRS07-full-draft-v1.pdf). Unfortunately, the submission fails to bring to SSC clustering what Nissim et al. could not bring to k-means-type clustering. Actually, the results look comparable from a high-level standpoint. Nissim et al approximate the Wasserstein distance DW (Lemma 5.6 in the longer version) to O (sigma k sqrt(d / n)) while the guarantee in DW for the submission is O (sigma k q log n). Why is it that DW does not decrease at least reasonably when n increases in Theorem 3.6 ? The data being i.i.d., I would expect this to happen. Furthermore, Lemma 3.5 (the key to Theorem 3.6) is very difficult to appreciate from the qualitative and quantitative standpoints. Drilling down into the conditions, we arrive at the following conclusions: for the probability to be 1-O(1), we *must* have n = O (exp (sqrt(d) / 2)) (first minus parameter in probability), which precludes too large samples. However, looking at the second minus parameter, Jensen's inequality brings n sum exp(-n_l / 400)\ geq k n exp (-n / (400k)). If k is, say, 10, then to have that last parameter say no more than 1 / 10, we must have exp(n / 4000) \geq 100 n, and thus n \geq 50 000. If d is not large enough, say d \leq 500, the bound is meaningless. And I am not even discussing the three assumptions of the lemma, that are just incredibly hard to capture to figure out a "competency map" of the Lemma that would give sufficient high-level constraints for the "1 - lots_of_things" to be not far from 1 (or even not negative...).Figuring out such "competency maps" is important and would prevent some "nasty" collateral consequences. For example, Lemma 3.3 is stated with $ \ phi ^ 2 \leq 1 / 802$. However, the constraint cost(hat(C)) \leq a cost (Copt) with a \leq (1-802phi ^ 2) / (800phi ^ 2) imposes, for the right hand side to be larger than 1 (otherwise the statement cannot hold), (1-802phi ^ 2) / (800phi ^ 2) \geq 1, and so... phi ^ 2 \leq 1 / 1602... The exponential mechanism with Gibbs sampling is original and interesting.The experiments give interesting results, * But * the following downsides apply: 1) the value of delta is a bit large, considering that delta = 1 / n is the trivial lower bound for which not doing any private mechanism still yields (epsilon, delta) differential privacy. 2) the value of n, d, k are clearly out of the applicable bounds for the theory, so we cannot use the experiments to validate in any way the ideas of the theory. 3) the values of epsilon in the plots is also too large, considering that in most applications, we would have epsilon \leq 1, thus truncating the plots to the [-1, 0] interval on the x-axis. In this case, the approach proposed by the authors clearly gives less compelling results.Finally, the lack of a conclusion, even short, summarising the pros and the cons, and opening future research directions is not a good thing either.Summary: the paper is original and its objectives are clear, but it needs some important technical revamping for the theory to be of broad relevance and be technically clear, in particular for the underlying assumptions to be unquestionable. The dependence in noise needs to be improved (or, at least, it must be shown why this large dependence in noise is absolutely necessary) to improve the paper's significance.


Dependent Multinomial Models Made Easy: Stick Breaking with the Pólya-Gamma Augmentation
Many practical modeling problems involve discrete data that are best represented as draws from multinomial or categorical distributions. For example, nucleotides in a DNA sequence, children's names in a given state and year, and text documents are all commonly modeled with multinomial distributions. In all of these cases, we expect some form of dependency between the draws: the nucleotide at one position in the DNA strand may depend on the preceding nucleotides, children's names are highly correlated from year to year, and topics in text may be correlated and dynamic. These dependencies are not naturally captured by the typical Dirichlet-multinomial formulation. Here, we leverage a logistic stick-breaking representation and recent innovations in Pólya-gamma augmentation to reformulate the multinomial distribution in terms of latent variables with jointly Gaussian likelihoods, enabling us to take advantage of a host of Bayesian inference techniques for Gaussian models with minimal overhead.

Summary: The paper presents a data-augmentation scheme for performing closed-form Gibbs sampling in models with Gaussian priors and multinomial likelihoods. The proposed scheme is based on introducing Polya-gamma random variables (one per observation) which allows Gaussianifying the multinomial likelihood and leading to closed-form Gibbs sampling updates. The proposed scheme is evaluated on a number of models with Gaussian priors and multinomial likelihood and is shown to demonstrate superior performance when compared to several baselines.Comments:Models with Gaussian priors and multinomial likelihoods are known to be troublesome when doing inference due to non-conjugacy. This paper presents a neat trick to "Gaussianify" the multinomial likelihood using an auxiliary variable construction based on the recently proposed Polya-gamma augmentation [3] by Polson and Scott (2012). This allows closed-form Gibbs samplingThe idea of using the Polya-gamma trick has been used in several papers recently for binary data (using logistic link) and count data (using negative binomial link). However, the muiltinomial case hasn't been explored much. I should also mention that the case of multinomial distribution was also considered in the original Polya-gamma paper by Polson and Scott (please see the appendix S6.3: http://arxiv.org/pdf/1205.0310v3.pdf) and used recently in [8]. This aspect should be discussed well before the proposed method is described.That said, this paper takes a somewhat different approach for handling the multinomial likelihood than the one presented in the appendix of [3]. Although the resulting equations here appear very similar to those obtained in [3], the proposed approach in this paper is more elegant in its construction (using a stick-breaking construction). In particular, this also enables block sampling whereas [8] which used the trick presented in [3] requires component-wise sampling. The experimental results corroborate the advantage of doing the block sampling. This makes the proposed construction very appealing.I also like the range of applications considered to demonstrate the usefulness and advantage of the proposed inference method over the alternatives.A minor point: I would be curious to see how much of a computational overhead does sampling the Polya-gamma variables introduces. At least, some discussion of this will be nice to have. Overall, although built on some standard pieces (Polya-gamma sampling, stick-breaking construction), the proposed inference method provides an easy way to do Gibbs sampling in non-conjugate models and is expected to be useful for a wide range of non-conjugate models with Gaussian priors and multinomial likelihood (as shown in some of the models presented in the paper).


Grammar as a Foreign Language
Syntactic constituency parsing is a fundamental problem in natural language processing and has been the subject of intensive research and engineering for decades. As a result, the most accurate parsers are domain specific, complex, and inefficient. In this paper we show that the domain agnostic attention-enhanced sequence-to-sequence model achieves state-of-the-art results on the most widely used syntactic constituency parsing dataset, when trained on a large synthetic corpus that was annotated using existing parsers. It also matches the performance of standard parsers when trained only on a small human-annotated dataset, which shows that this model is highly data-efficient, in contrast to sequence-to-sequence models without the attention mechanism. Our parser is also fast, processing over a hundred sentences per second with an unoptimized CPU implementation.

This paper proposes a method to learn a syntactic constituency parser using an attention-enhanced sequence-to-sequence model. The paper claims that it obtains state-of-the-art results on WSJ and performs comparatively to Berkeley parser.The problem of syntactic parsing is an important one in NLP, and is a fundamental pre-processing step for more advanced tasks such as named entity extraction or relation extraction. Therefore, any improvement on syntactic parsing is likely to have a significant impact on the community. To this end the application of LSTM-based sequence-to-sequence coding is both timely and important.Additional strong points of the proposed parser are its speed, and domain adaptability.However, I have several concerns that I would wish the authors could further clarify as mentioned below.The title of the paper and its relevance to the current work is somewhat unclear. I do understand that sequence-to-sequnce mapping has been previously applied for machine translation, but what exactly do you mean by "grammar as a foreign language"? The relevance of the title to the proposed method is never explicitly discussed in the paper. I guess you wanted to hint that you could "learn" the grammar of a language using sequence-to-sequence models. But the evaluation tasks are all monolingual adaptation task and not cross-lingual parsing tasks. There are several statements in the paper that require further clarifications or analysis. For example, it is mentioned that replacing all pos tags by XX gave an F1 improvement of 1 point. Why is this? Removing POS info means you are loosing many important features that have been found to be useful for syntactic parsing.What is the significance that the constructed parser matches Berkeley parser with 90.5 F1? Should not it be the case that the comparison should be done against an external test dataset and not the parser that generated the train data in the first place?The degree of novelty w.r.t. machine learning or deep learning is minimal in the paper considering that it is using a previously proposed sequence-to-sequence encoding method based on LSTM for parsing. On the other hand, the claims of the paper and application task (parsing) is more relevant to the NLP community. The paper would be much appreciated and recognized at an NLP conference such as ACL or EMNLP than at NIPS. Therefore, it would be more suitable to submit this work at an NLP related conference.The problem of domain adaptation of parsers has been considered in the NLP community and arguably the most famous benchmark for evaluating this would be SANCL-2012 "parsing the web" task (see Overview of the 2012 Shared Task on Parsing the Web by Petrov and McDonald 2012). Unfortunately, this line of work is not discussed nor evaluated against as a benchmark dataset in the current paper, which makes it difficult to justify the claims about domain adaptability of the proposed method.The robustness of the parser is also a concern. It is mentioned that all 14 cases where the parser failed to produce a balanced tree are fragments or not ending with proper punctuations. Unfortunately, majority of the text found in social media such as twitter would be such cases. It is not discussed in the paper how you could address this issue (in particular that you claim the domain adaptability of the parser) in a more systematic manner than simply adding brackets to balance it.


Efficient Thompson Sampling for Online Matrix-Factorization Recommendation
Matrix factorization (MF) collaborative filtering is an effective and widely used method in recommendation systems. However, the problem of finding an optimal trade-off between exploration and exploitation (otherwise known as the bandit problem), a crucial problem in collaborative filtering from cold-start, has not been previously addressed. In this paper, we present a novel algorithm for online MF recommendation that automatically combines finding the most relevant items with exploring new or less-recommended items. Our approach, called Particle Thompson sampling for MF (PTS), is based on the general Thompson sampling framework, but augmented with a novel efficient online Bayesian probabilistic matrix factorization method based on the Rao-Blackwellized particle filter. Extensive experiments in collaborative filtering using several real-world datasets demonstrate that PTS significantly outperforms the current state-of-the-arts.

The authors propose an online matrix completion algorithm that employs particle filters to represent the posterior distribution over user/item factors. Thompson sampling is employed to recommend items according to this approximated posterior. Particle are then resampled, but only the vectors of item and user corresponding to the most recently received rating.A theoretical bound on the regret is provided, and the method is analyzed on synthetic data, and compared against various baseline strategies on 5 real datasets. The method outperforms all but one baseline on all the datasets (and the best baseline is considered unrealistic but see question below). The value of this method hinges on the assumption that uncertainty of *both* user and item factors is necessary to make reasonable recommendations in the cold-start scenario, which is illustrated in Figure 4c.Overall, this paper may be a solid contribution but I have some concerns that need to be resolved:1. In 4/5 of the real datasets, the proposed method is either comparable orworse than the strategy of picking the most popular item (restricted to those recommended for that user). This is labelled as an 'unrealistically optimistic baseline' as it is not possible to know global popularity beforehand. However, it certainly seems possible to estimate global popularity on a training set (akin to how ICF estimates item factors on a training set) and evaluate on a test set. This also seems very possible in a real-world setting (collect information on global popularity before deploying your recommendation system). I think a comparison with such a strategy is warranted.2. How were K=2 and 30 particles chosen? How does the algorithm scale and perform for larger K? Do you need many more particles to represent posteriors over larger-dimensional objects for these datasets?3. The authors acknowledge that ICF performance is sensitive to the amount of training data used. A plot showing how ICF performs against PTS/PTS-B as a function of training proportion would be helpful here. Some other minor points:1. It's mentioned that Thompson sampling is used to recommend the item, but in Algorithm 1, the item chosen to be recommended is simply \argmax_j \tilde{U}_i^T\tilde{V}_j. Is the ``sampling'' part simply coming from the sampling of \tilde{U} and \tilde{V} using the particles?2. The method performs slightly worse in an offline setting than standard PMF (0.7925 vs. 0.7718) - is there an explanation for why?3. The following citable work (obtained via google search of 'matrix completion bandit problem') seems very relevant, yet is not cited:Linear Bandits, Matrix Completion, and Recommendation Systems. Madeleine Udell and Reza Takapoui. NIPS Workshop on Large Scale Matrix Analysis and Inference, 2014.


Sparse Local Embeddings for Extreme Multi-label Classification
The objective in extreme multi-label learning is to train a classifier that can automatically tag a novel data point with the most relevant subset of labels from an extremely large label set. Embedding based approaches attempt to make training and prediction tractable by assuming that the training label matrix is low-rank and reducing the effective number of labels by projecting the high dimensional label vectors onto a low dimensional linear subspace. Still, leading embedding approaches have been unable to deliver high prediction accuracies, or scale to large problems as the low rank assumption is violated in most real world applications. In this paper we develop the SLEEC classifier to address both limitations. The main technical contribution in SLEEC is a formulation for learning a small ensemble of local distance preserving embeddings which can accurately predict infrequently occurring (tail) labels. This allows SLEEC to break free of the traditional low-rank assumption and boost classification accuracy by learning embeddings which preserve pairwise distances between only the nearest label vectors. We conducted extensive experiments on several real-world, as well as benchmark data sets and compared our method against state-of-the-art methods for extreme multi-label classification. Experiments reveal that SLEEC can make significantly more accurate predictions then the state-of-the-art methods including both embedding-based (by as much as 35%) as well as tree-based (by as much as 6%) methods. SLEEC can also scale efficiently to data sets with a million labels which are beyond the pale of leading embedding methods.

The submission describes a scalable method for large multi-label prediction problems with very long-tailed label distributions.The method first learns a nonlinear embedding of the label (output) vectors that preserves local distances.A linear map is then learned from features (inputs) into this embedding space.At test time, the linear map is evaluated to predict a point in the embedding space, and then the labels are predicted via KNN in this space (i.e., the predicted label vector is the sum of those of training examples with similar embeddings).A few tricks are used to achieve scalability and good generalization: namely, clustering is used to reduce the complexity of KNN, and averaging over the randomness of clustering results is used to mitigate the instability of high-dimensional clustering.Experiments show that this method is faster to evaluate, more scalable at training, and generalizes better than competing methods.Significance: The experimental results are the strongest part of the paper, which show convincingly better results in multiple aspects compared to existing state-of-the-art methods.However, the method is a bit complex compared to other approaches, in the sense that it is a pipeline with a few steps and a number of parameters to tune.This makes it a bit less likely to catch on and have a big practical impact.Originality: The basic solution approach builds on previous embedding-based methods, and as such, the component parts have mostly appeared elsewhere previously.The most novel part about the method seems to be the idea to use a nonlinear, local-distance-preserving embedding as opposed to a linear, global embedding of the labels, but this is a fairly straightforward innovation.Clarity: Much can be done to improve the clarity of the submission.First, the paper suffers from information overload in a few places, making it difficult to tell what is important and what is not.As mentioned elsewhere, the theoretical analysis, though interesting, seems to be of little relevance to the actual method, so perhaps this section should be abridged.The experiments are very thorough, which is good, but in this section and throughout the paper, I would have preferred it if the discussion had focused on discussing the most important points well rather than trying to cram in as many points as possible.Quality: Overall, this is a well-thought-out method to solve a real-world problem well.Thoughtful concern is given to the many details required to make a system performant and scalable on real data.The experiments are also thorough with strong results.The theoretical analysis also seems very sophisticated, but unfortunately seems like a tangent with little relevance to the rest of the work.The weakest part of the work is that the novelty of the method is relatively low.Other comments:From a technical perspective, I was a bit disappointed at the disconnect between the proposed optimization for the embedding and the method used to solve it.The idea of learning an embedding that is nonlinear as a map from labels, but linear as a map from features is very interesting, because the linearity-from-features constraint serves to regularize an embedding which might otherwise be too powerful to produce meaningful results.Unfortunately, the algorithm used to actually optimize this objective learns the embedding completely ignoring the constraint that it must be realizable as a linear map from the features.So, the obtained solution effectively optimizes an objective which differs from the desired objective (3) in a very important way.This is also one reason why the ERM bound fails to provide justification for the method: there is little reason to believe that the obtained solution should be close to the optimum of (3).A related concern is that preserving local distances alone is not sufficient to ensure good accuracy of KNN, since a nonlinear embedding can easily preserve local distances while also pushing points into neighborhoods in which they do not belong.This is why methods such as Large Margin Nearest Neighbor exist.Unfortunately, such methods tend to be computationally expensive, so I worry that taking this into account would spoil the computational efficiency of the method.These concerns make it a little surprising that the method works so well in practice.POST-REBUTTAL COMMENTS:The rebuttal did not significantly change my opinion.I am positive overall about the work, mostly due to the experimental results, but I still have a few concerns.Basically, the presentation makes it seem as though certain decisions are motivated by rigorous theory, but this is not really the case.The two-step optimization procedure used in practice differs in a critically important way from the theoretically well-motivated joint optimization.The fact that the objective does not properly preserve neighborhoods also makes it surprising that the method works well in practice.If I were to revise the manuscript, I would put less emphasis on the marginally-related theoretical results and more emphasis on giving plausible explanations, empirical or otherwise, for these counterintuitive phenomena.


Minimum Weight Perfect Matching via Blossom Belief Propagation
Max-product Belief Propagation (BP) is a popular message-passing algorithm for computing a Maximum-A-Posteriori (MAP) assignment over a distribution represented by a Graphical Model (GM). It has been shown that BP can solve a number of combinatorial optimization problems including minimum weight matching, shortest path, network flow and vertex cover under the following common assumption: the respective Linear Programming (LP) relaxation is tight, i.e., no integrality gap is present. However, when LP shows an integrality gap, no model has been known which can be solved systematically via sequential applications of BP. In this paper, we develop the first such algorithm, coined Blossom-BP, for solving the minimum weight matching problem over arbitrary graphs. Each step of the sequential algorithm requires applying BP over a modified graph constructed by contractions and expansions of blossoms, i.e., odd sets of vertices. Our scheme guarantees termination in O(n 2 ) of BP runs, where n is the number of vertices in the original graph. In essence, the Blossom-BP offers a distributed version of the celebrated Edmonds' Blossom algorithm by jumping at once over many sub-steps with a single BP. Moreover, our result provides an interpretation of the Edmonds' algorithm as a sequence of LPs.

This paper introduces two polytime algorithms for minimumweight perfect matching using a sequence of linear programs orruns of max-product BP (the factor graphs for which areinspired by connections to the LPs). To obtain the polytimealgorithm using BP, the paper makes use of a result fromrecent prior work (Park and Shin, 2015): under specificconditions, for certain pairs of a factor graph and an LP,running max-product BP on the factor graph converges to thesolution of the LP. The paper proves that the iterativealgorithm uses at most O(n^2) iterations where n is the numberof vertices in the graph.The authors speculate that theirwork may spur further study of BP for MAP on more generalgraphical models. Though the paper does not explicitly stateany ideas in this direction, it seems like a furtile area. Aswell, the new LP algorithm provides an interesting contrastwith Edmonds' Blossom algorithm, which (in its original form)required exponentially many constraints. Overall, it's clearlywritten, original work that, while not the first polytimealgorithm for min-weight perfect matching (they reserve thattitle for (Chandrasekaren et al., 2012)), is likely thesimplest.  It seems the paper would benefit from a journal-stylepresentation: one which is more leisurely and examplefilled. Given space constraints the paper did a good job ofdelegating proofs of certain results to supplementarymaterial. However, the main paper would benefit greatly if theexamples from Appendix F could be included in a figure, andrepeatedly referenced through the description of thealgorithm.  On the one hand, this paper can be viewed as a niceapplication of Park and Shin (2015)'s BP/LP result. On theother, it stands as a unique example (to the best of myknowledge as well as claimed by the authors) of solving an ILPby repeated calls to BP.  This paper does not lack substance, but it would have beennice to see the algorithm put to use on some real matchingproblems. As the linear programming community knows well,polynomial time algorithms are sometimes less practical thantheir exponential time counterparts in the real world.  Misc:- line 155: Linear Programming --> Linear Program- line 190: this note deserves a one line explanation- line 196: initially confusing where T came from- line 258: if the proof of Theorem 3 will be left in theappendix, it would help to give some intuition for why thehalf-integral solution is obtained.- line 032 (appendix A): marginal beliefs --> max-marginals- A more detailed contrast with Chandrasekaren et al. (2012)seems appropriate


Robust Regression via Hard Thresholding
We study the problem of Robust Least Squares Regression (RLSR) where several response variables can be adversarially corrupted. More specifically, for a data matrix X ∈ R p×n and an underlying model w * , the response vector is generated as y = X T w * + b where b ∈ R n is the corruption vector supported over at most C · n coordinates. Existing exact recovery results for RLSR focus solely on L 1 -penalty based convex formulations and impose relatively strict model assumptions such as requiring the corruptions b to be selected independently of X. In this work, we study a simple hard-thresholding algorithm called TORRENT which, under mild conditions on X, can recover w * exactly even if b corrupts the response variables in an adversarial manner, i.e. both the support and entries of b are selected adversarially after observing X and w * . Our results hold under deterministic assumptions which are satisfied if X is sampled from any sub-Gaussian distribution. Finally unlike existing results that apply only to a fixed w * , generated independently of X, our results are universal and hold for any w * ∈ R p . Next, we propose gradient descent-based extensions of TORRENT that can scale efficiently to large scale problems, such as high dimensional sparse recovery. and prove similar recovery guarantees for these extensions. Empirically we find TOR-RENT, and more so its extensions, offering significantly faster recovery than the state-of-the-art L 1 solvers. For instance, even on moderate-sized datasets (with p = 50K) with around 40% corrupted responses, a variant of our proposed method called TORRENT-HYB is more than 20× faster than the best L 1 solver.
"If among these errors are some which appear too large to be admissible, then those equations which produced these errors will be rejected, as coming from too faulty experiments, and the unknowns will be determined by means of the other equations, which will then give much smaller errors."
A. M. Legendre, On the Method of Least Squares. 1805.

This paper proposes and analysis a new algorithm for the robust regression problem. Robust regression problem is an old problem, related to outliers detection.Introduction ---------------The robust regression problem is defined in Eq. (2). I think that the authors should slightly rephrase their definition as Eq. (2) is one possible "modern" definition of the robust least square problem. Authors also claims that robust regression and sparse regression are "completely different". The only difference if the sparse assumption on the coefficients vector w. The robust regression has a much more specific model on the noise, but can also be a sparse regression (as studied in Sec. 5). Again, I think that the authors should rephrase this part to avoid miss-understanding.My major remark here, is a lake of state of the art on robust regression/outliers detection. Indeed, only very recent work are mentioned, but this a very old problem, and some classical methods such as L1 regression (in the sense of min || y - X w ||_1 ) should be mentioned here, and used as a reference.Sec 2 ------The problem formulation is clear, and the noise is specified thanks to two terms : - a classic gaussian noise - an unbounded "sparse" noise.However, this formulation is not the "only one" possible. Maybe one reference on some Bayesian model for the noise on robust regression would be welcome here, in order to have a quite complete point of view on robust regression.Moreover, I think that the authors could reformulate clearly the robust least square regression problem as an optimisation problem such as (there is some others possibilities) min_{w,b} 0.5 ||y - Xw - b||^2 + \lambda ||b||_0Sec. 3 -------This section introduces the "Torrent" algorithms studied in Sec. 4. This algorithms made me think of a kind of "matching pursuit": the selection of the active set is made directly on the residual instead on its correlation with the matrix X. Torrent-FC being the "OMP" and Torrent-GD the "classical" MP.The definition of Hard thresholding suits well with the problem, but is a kind of counter-intuitive compared to the classical hard thresholding used for sparse regression.  Sec. 4 ------This section presents the theoretical results of the proposed algorithm, and then is the main theoretical contribution of the paper. I have no particular remark on this part.Sec. 5 -------This section extends previous results to the "Robust sparse regression problem". Following my remark on Sec. 2,I think that at this point, the author should re-formulate the problem asmin_{w,b} 0.5 ||y - Xw - b||^2 + \lambda ||b||_0 + \mu ||w||_0 Sec. 6 -------The experimental part if ok, but the only method the authors use for comparison is L1-DALM which solve only the sparse robust regression problem, not the robust regression problem without the sparse assumption. So the comparison is not fair, as the goal of the two approached is not the same. I also think that the author must compared their algorithm to a classic L1-regression problem, which is the state of the art of robust regression (and also a L1-L1 approach for the robust sparse regression problem).


Segregated Graphs and Marginals of Chain Graph Models
Bayesian networks are a popular representation of asymmetric (for example causal) relationships between random variables. Markov random fields (MRFs) are a complementary model of symmetric relationships used in computer vision, spatial modeling, and social and gene expression networks. A chain graph model under the Lauritzen-Wermuth-Frydenberg interpretation (hereafter a chain graph model) generalizes both Bayesian networks and MRFs, and can represent asymmetric and symmetric relationships together. As in other graphical models, the set of marginals from distributions in a chain graph model induced by the presence of hidden variables forms a complex model. One recent approach to the study of marginal graphical models is to consider a well-behaved supermodel. Such a supermodel of marginals of Bayesian networks, defined only by conditional independences, and termed the ordinary Markov model, was studied at length in [6]. In this paper, we show that special mixed graphs which we call segregated graphs can be associated, via a Markov property, with supermodels of marginals of chain graphs defined only by conditional independences. Special features of segregated graphs imply the existence of a very natural factorization for these supermodels, and imply many existing results on the chain graph model, and the ordinary Markov model carry over. Our results suggest that segregated graphs define an analogue of the ordinary Markov model for marginals of chain graph models. We illustrate the utility of segregated graphs for analyzing outcome interference in causal inference via simulated datasets.

The authors try to tackle the problem when bidirected edge (<->) and undirected edge (-) meet in a loopless mixed graph. They propose to construct a new mixed graph called segregated graph (SG) (where bidirected edge (<->) and undirected edge (-) never meet) which preserves all conditional independencies on the observed variables. I like the originality of the idea. I am somewhat disappointed by the clarity of the paper, which affects the quality too. Also the significance of the problem considered is not well addressed. Here I give more details on each criterion: *Clarity: I like the authors' explanation of the big picture before going to details, which helps to clarify to some degree. However, once we check the details, it becomes less clear caused by typos and undefined/undeclared symbols. It would be much easier to read if the authors follow notations in any of the well known (and recent) publications on this topic such as [4], [1*], and [2*].*Quality: it is dragged down by the level of clarity. Also I notice that if we move most of the proofs to the supplementary file, the paper would have much more spare space (2-3 pages perhaps). This suggests much more content can be added to the paper. *Originality: good. *Significance: the importance of the problem is only lightly touched (line 128-129) with a bit further discussion on the implication in conclusions. Missing references: [1*] Sadeghi, K. and Lauritzen, S. (2014). Markov properties for mixed graphs. Bernoulli 20, 676-696. MR3178514 [2*] Wermuth, N. (2011). Probability distributions with summary graph structure. Bernoulli 17, 845-879. MR2817608After reviewers' dissuasion and seeing authors' response, I've increased the score by one step.


Fast and Memory Optimal Low-Rank Matrix Approximation
In this paper, we revisit the problem of constructing a near-optimal rank k approximation of a matrix M ∈ [0, 1]   m×n under the streaming data model where the columns of M are revealed sequentially. We present SLA (Streaming Low-rank Approximation), an algorithm that is asymptotically accurate, when
is the (k + 1)-th largest singular value of M . This means that its average mean-square error converges to 0 as m and n grow large (i.e.,
with high probability, whereM (k)  and M (k) denote the output of SLA and the optimal rank k approximation of M , respectively). Our algorithm makes one pass on the data if the columns of M are revealed in a random order, and two passes if the columns of M arrive in an arbitrary order. To reduce its memory footprint and complexity, SLA uses random sparsification, and samples each entry of M with a small probability δ. In turn, SLA is memory optimal as its required memory space scales as k(m+n), the dimension of its output. Furthermore, SLA is computationally efficient as it runs in O(δkmn) time (a constant number of operations is made for each observed entry of M ), which can be as small as O(k log(m) 4 n) for an appropriate choice of δ and if n ≥ m.

Let $M^k$ be the best rank $k$ approximation to $M$.1. The right way to evaluating the approximation error is contrasting $|| M - \tilde{M}^k ||$ with $|| M - M^k||$, rather than contrasting $|| M^k - \tilde{M}^k ||$ to $mn$.That $|| M^k - \tilde{M}^k || = o(mn)$ does not indicate convergence! E.g. Let m=n and $||M|| = O(n^2)$ by this paper's assumption. Assume the singular values decays slowly in that $\sigma_{max}^2 = c \sigma_{min}^2$ for a constant $c$. Then $||M^k||_F^2 < k \sigma_{max}^2 = k c \sigma_{min}^2 < k c n^{-1} ||M||_F^2 = O(kn) = o(n^2)$. This indicates that even the all-zero matrix satisfies this paper's error bound: $|| 0 - M^k ||_F^2 = o(n^2)$. 2. This paper proves an IMPOSSIBLE convergence bound. Assume that the 1st to the 2k-th singular values are equal. Any among the first 2k principal components cannot be distinguished from each other [2]. Thus $\tilde{M}^k$ can be entirely different from $M^k$, which means the impossibility of convergence. (In fact, in this case, $\tilde{M}^k$ converges to $M^k$ with ZERO measure.) Interestingly, the authors tell us their streaming PCA converges to $M^k$. 3. Stronger results have been established by Boutsidis & Woodruff [1].[1] arXiv:1504.06729v1, April 25, 2015. Communication-optimal Distributed Principal Component Analysis in the Column-partition Model. Boutsidis, Christos, Woodruff, David P.[2] Topics in matrix analysis. RA Horn, CR Johnson.


Sample Complexity of Learning Mahalanobis Distance Metrics
Metric learning seeks a transformation of the feature space that enhances prediction quality for a given task. In this work we provide PAC-style sample complexity rates for supervised metric learning. We give matching lower-and upper-bounds showing that sample complexity scales with the representation dimension when no assumptions are made about the underlying data distribution. In addition, by leveraging the structure of the data distribution, we provide rates fine-tuned to a specific notion of the intrinsic complexity of a given dataset, allowing us to relax the dependence on representation dimension. We show both theoretically and empirically that augmenting the metric learning optimization criterion with a simple norm-based regularization is important and can help adapt to a dataset's intrinsic complexity yielding better generalization, thus partly explaining the empirical success of similar regularizations reported in previous works.

Summary In this article the authors present a theoretical analysis of non-regularized supervised metric learning formulations. They study two frameworks. On the one hand, the distance-based framework considers algorithms that optimize a distance w.r.t. class label information. On the other hand, the classifier-based framework considers algorithms which optimize a metric for a specific prediction or a ranking task.For both frameworks, the authors provide PAC style bounds (sample complexity bounds) on the difference between the true risk and the empirical risk of the learned metric. They also analyze the difference between the true risk of the learned metric and the true risk of the best in-class metric and discuss the the necessity of a strong dependency on the dimensionality of the data. In the last part of the paper, the authors consider some regularized formulation using the Frobenius norm as aregularizer, they propose some refinement of their results in this context and argue that adding a regularization term provides better guarantees. The authors propose finally an empirical evaluation of two well known metric learning approaches (namely ITML and LMNN) showing that adding a regularization term is indeed beneficial. The paper is relatively clear and well written. The results are new and provide new theoretical understanding for metric learning . Comments.-Correct the proof of Lemma 2 to make it independent from the dimension. You should add some comments to make a link between Th1 and Th2. In the current form of the paper, they are presented separately and with formulations that do not allow the reader to directly get the consequences on the dependence on D.-Mention that the results of Jin et al. (NIPS 2009) can be extended to replace the dependency on d by a term based on the data concentration (term B in author's paper), for any distributions and convex regularized formulations.This can be done by using the techniques of Bousquet & Elisseeff, JMLR 2002 - note that the convexity of the formulation is mandatory for applying algorithmic stability. +Bellet et al. Metric Learning. Morgan & Claypool publishers. 2015. (chapter 8)-One of the claim of the paper is to generalize some previous results to possibly non convex loss. However, in this context there is no guarantee to find the best model (matrix) minimizing the risk which makes the results a bit non informative on what can be done in practise.Do not forget that even for convex formulation, taking into all the quadratic constraints involved by classical metric learning formulations is heavy and in general some heuristics are used to reduce the number of constraints. One interesting perspective would be to know if the dependency on D impacts more severely non convex non regularized formulations.-The authors have focused on the dependence over the representation dimension D, however it could be interesting to consider also the true rank of the matrix M rather than the data dimension. This would allow to make some relationships with metric learning based on Cholesky decomposition of the learned metric and methods making use of some low rank regularization. This point should be at least discussed in the perspectives.-Considering, the classifier-based approach, the bibliography lacks a reference on methods learning a linear classifier from good similarity functions, even if this paper adresses other settings. +Balcan et al. Improved Guarantees for Learning via Similarity Functions. COLT 2008. +Bellet et al. Similarity Learning for Provably Accurate Sparse Linear Classification. ICML 2012. This aspect is also studied in the paper of Guo and Ying cited by the authors.Overall, the results are interesting and I think that providing theoretical guarantees on classifiers making use of the result of a known metric is very important and results in that direction must continued.The results are based on covering and fat shattering dimensions arguments, is there any perspective to use other frameworks (Rademacher-based for example?)-About the experiments.Even though they do not consider noisy data in their analysis, the authors create noise augmented datasets to see the influcence of the regularization. The conclusion of the experiments is that regularization can help to deal with noisy datasets or in other words they point out that the well known problem of overfitting can be partly adressed by choosing a good regularizer which is known fact. The scaling for the x-axis is different for the 3 figures, what is the reason for that? In particular, one can wonder what happens after 120/150 for Iris and Ionosphere? For the comparison with ITML, it could be interesting to have the accuracy obtained by directly using the rank-1 matrix alone in the distance to compare the influence of the regularizer. It seems that the choice of this matrix may do a lot of the "job". The experiments lack of datasets - 3 is very small and the number of dimensions is relatively low - and lack of methods for comparison. We can also wonder why no experiment on any classifier-based setting has been made. There also exist other datasets with higher dimensions in the UCI datasets, you can also check here : +Y. Shi et al.:Sparse Compositional Metric Learning. AAAI 2014: 2078-2084. The experiments would have gained in interest, if they authors have presented the evolution of the bounds with the data and with the dimension.  Other comments:-line 075: the authors consider binary classification, however the loss considered below allows to consider a multiclass setting and some data sets in the experiments are multiclass. -I am wondering why in the definition of the true risk of the classifier-based approach, the margin is chosen at 1/2, while for the empirical risk the margin can be fixed to any value between 0 and 1/2. -line 219: "do now know" -> "do not know" -in the proof of Lemma 4, the notion of covering is used extensively. Is it possible to extend the results to other regularization norms?


Explore no more: Improved high-probability regret bounds for non-stochastic bandits
This work addresses the problem of regret minimization in non-stochastic multiarmed bandit problems, focusing on performance guarantees that hold with high probability. Such results are rather scarce in the literature since proving them requires a large deal of technical effort and significant modifications to the standard, more intuitive algorithms that come only with guarantees that hold on expectation. One of these modifications is forcing the learner to sample arms from the uniform distribution at least Ω( √ T ) times over T rounds, which can adversely affect performance if many of the arms are suboptimal. While it is widely conjectured that this property is essential for proving high-probability regret bounds, we show in this paper that it is possible to achieve such strong results without this undesirable exploration component. Our result relies on a simple and intuitive loss-estimation strategy called Implicit eXploration (IX) that allows a remarkably clean analysis. To demonstrate the flexibility of our technique, we derive several improved high-probability bounds for various extensions of the standard multi-armed bandit framework. Finally, we conduct a simple experiment that illustrates the robustness of our implicit exploration technique.

ADDED after author response:Thank you for the clarifications. It would be great if you could include some of the points about implicit vs. explicit exploration and about choice of parameters in the two settings in the final version.======The work is motivated by the observation that the explicit exploration term often used in multi-armed bandit algorithms when selecting the next arm can adversely affect performance, especially in situations where some arms are "obviously" suboptimal. The authors instead propose to use the implicit exploration (IX) idea of Kocak et al [NIPS-2014] and show that the corresponding loss function with implicit exploration (a) simplifies the regret bound analysis and (b) improves the constant factor in known regret bounds for many variants of the basic MAB problem.      The paper is interesting, the writing is clear, and the math seems solid (although I didn't check all proofs carefully). Table 1 neatly summarizes the main theoretical results. I enjoyed reading the paper but have only lukewarm feelings of acceptance because:      1. Novelty and significance: The novel part here is the application of the known IX idea to a number of known algorithms, and their analysis. The results in table 1 are clearly improvements, but "only" by small constant factors. I'm not sure how significant is the result in advancing the field or how much practical impact they would have.      2. Implicit vs. Explicit exploration: The basic tenet here is that explicit exploration is undesirable, which I agree with. There are at least two things that make it undesirable. First is that with explicit exploration comes a tradeoff parameter (how much to explore, how much to exploit) that must often be tuned in practice based on the domain at hand. The proposed algorithm, however, effectively has this parameter as well (the gamma in the denominator) and thus doesn't resolve this issue. Second, as noted in the abstract, explicit exploration makes algorithms sample the losses of every arm at least Omega(sqrt(T)) times over T rounds, even when many arms are "obviously suboptimal". The (implicit) claim is that the proposed algorithm is less wasteful on arms that are "obviously suboptimal". However, this part of the theory has not been explored in the paper. Table 1 doesn't address this. Reading the abstract, I was expecting to see some result of the form: if some fraction of arms is "obviously suboptimal" then the IX algorithm will provably waste less resources on them. Having a result along these lines would have made the paper significantly stronger.      Overall, I like the paper but have some reservations.


Deeply Learning the Messages in Message Passing Inference
Deep structured output learning shows great promise in tasks like semantic image segmentation. We proffer a new, efficient deep structured model learning scheme, in which we show how deep Convolutional Neural Networks (CNNs) can be used to directly estimate the messages in message passing inference for structured prediction with Conditional Random Fields (CRFs). With such CNN message estimators, we obviate the need to learn or evaluate potential functions for message calculation. This confers significant efficiency for learning, since otherwise when performing structured learning for a CRF with CNN potentials it is necessary to undertake expensive inference for every stochastic gradient iteration. The network output dimension of message estimators is the same as the number of classes, rather than exponentially growing in the order of the potentials. Hence it is more scalable for cases that involve a large number of classes. We apply our method to semantic image segmentation and achieve impressive performance, which demonstrates the effectiveness and usefulness of our CNN message learning method.

Given CRF potentials, one can write down the marginal predictions after only passing the initial messages from factors to variables. This paper incorporates that parametric form into a neural network, and fits it directly to minimize training error. This architecture appears to be empirically successful on a meaningful benchmark. I found the framing of the work partially misleading: As far as I can tell, the cost function reported doesn't care about structured prediction just pixel-wise errors, and no CRF model is actually fitted.To me "structured CRF prediction" means that there is a joint distribution over the labels given an input. Given such a model, it would be possible to fantasize joint plausible configurations of the labels, or report the jointly most probable labeling.I don't think the "Intersection over Union" score cares if the vector of labels is jointly plausible. The way to maximize this score is to predict the most probable label for each pixel independently. Learning a CRF model would learn a joint distribution, but here predictive performance, which doesn't care about the joint distribution, is maximized, not a CRF model. It isn't even clear to me if the "messages" in this paper will always correspond to realizable factors(?), let alone likely ones.The work is certainly interesting. The idea is a neat one, and seems to be an excellent heuristic for setting the parametric form of the output layer of a neural network to get good performance. I simply found some of the framing misleading and genuinely confusing on first reading.  line 332: "One advantage of message learning is that we are able to explicitly incorporate the expected number of inference iteration into the learning procedure." -- However, only one iteration is ever tried, and I presume implemented. I'm not convinced the details have been worked out or investigated enough to make this claim. Typo: line 54: massage -> messageThere's quite a few incorrect word endings, usually singular/plural disagreement. I lost motivation to note them down after noticing a spell checker hadn't been run: line 66: potenials line 120: sematic line 265: convlutional


Alternating Minimization for Regression Problems with Vector-valued Outputs
¬¸»² ©» ©·´´¸¿ª» µ© © µ î ã°²÷ò Ø±©»ª»®ô «¾¬®¿½¬·²¹ ¬¸» ¬©± »¯«¿¬·±² ¹·ª»ae § ï § î ã øÈ ï È î ÷ â © å ï · ²ò Ì¸¿¬ ·ô ¿ ±±² ¿ ©»¸¿ª» ² ¼ ¿³°´»ô ©» ©·´® »½±ª»® © »¨¿½¬´ § ¾ § ±´ª·²¹ ¬¸» ¿¾±ª» §¬»³ ±º´·²»¿® »¯«¿¬·±²ÿ Ñ«® ¬± § »¨¿³°´» ³±¬·ª¿¬» ¬¸» º«²¼¿³»²¬¿´¯«»¬·±² ¬¸¿¬ ¬¸·°¿°»® ¿²©»®ae¸±© ³«½¸½¿² ©» ·³°®±ª» ÑÔÍ ¾ § »¨°´±·¬·²¹ ²±·» ½±®®»´¿¬·±²á Ô»¬ « ³¿µ» ¬¸» »¨¿³°´» ³±®» ®»¿´·¬·½ ¾ § ¿«³·²¹ ¬¸» ³±¼»´ae § ã È © õ å ï · ²ô ©¸»®» § î Î · ¿ ª»½¬±® ±º ³ ®»°±²»ô »¿½¸»´»³»²¬ ±º È î Î · ¿³°´»¼ ·ò·ò¼ò º®±³ ¬¸» ¬¿²¼¿®¼ Ù¿«·¿² ¿²¼ ²±·» ª»½¬±® · ¼®¿©² º®±³ Ò øðå ÷ò ß ½±®±´´¿® § ±º ¬¸» ³¿·² ®»«´¬ ·² Í»½¬·±² î ¸±© ¬¸¿¬ ÓÔÛ ø»» øí÷÷ ·³°®±ª» «°±² ¬¸» ÑÔÍ°¿®¿³»¬»® »®®±® ¾±«²¼ ¾ § ¿ º¿½¬±® ±º Û®®±® ãÛ®®±® ï ÷ã³ î ò Ì¸· º¿½¬±® ½¿² »¿·´ § ¾» »»² ¬± ¾»´¿®¹»® ¬¸¿² ï ¾ § «·²¹ Ý¿«½¸ §óÍ½¸©¿®¦ ·²»¯«¿´·¬ §ae ï ÷ã³ î ã ø ÷ø ï ÷ã³ î ø ïã ÷ î ã³ î ã ïô ©¸»®» ¾» ¬¸» ¶ó¬¸´¿®¹»¬ »·¹»²ª¿´«» ±º ò Ì¸» ·²»¯«¿´·¬ § · ¬·¹¸¬ ©¸»² ã ½ã º±® ±³» ½±²¬¿²¬ ½ò Ì¸¿¬ ·ô ©¸»² ã ½× ©¸·½¸¸±´¼ ¬®«» ·ºº ¬¸» ²±·» ·² »¿½¸®»°±²» · ³«¬«¿´´ § ·²¼»°»²¼»²¬ ¿²¼¸¿ ¿³» ª¿®·¿²½»ò Ì¸» ³±®» ¼»°¿®¬ º®±³ ¾»·²¹ ½ ×ô ¬¸»´¿®¹»® ¬¸» ·³°®±ª»³»²¬ º¿½¬±®ò Ú±® »¨¿³°´»ô ½±²·¼»® ³ ã î ½¿» ¿¹¿·²ô ¾«¬ ®¿¬¸»® ¬¸¿² ï ã î ô ©»¸¿ª»¸·¹¸´ § ½±®®»´¿¬»¼ Å ï î Ã ©·¬¸½±ª¿®·¿²½» ³¿¬®·¨ã
Í»» ß°°»²¼·¨Ý º±® ¼»¬¿·´»¼°®±±º ±º ¾±¬¸±º ¬¸» ®»«´¬ ¹·ª»² ¿¾±ª»ò
É» ²±© ½±³¾·²» ¬¸» ¿¾±ª» ¹·ª»² ±¾»®ª¿¬·±² ©·¬¸Ì¸»±®»³ î ¬± ±¾¬¿·² ±«® »®®±® ¾±«²¼ º±® ß´¬Ó·²°® ±½»¼«®»ò Ë·²¹ ¿ ´·¹¸¬´ § ¬®±²¹»® ª»®·±² ±º Ì¸»±®»³ ïëô ©» ½¿² ¿´± ±¾¬¿·² ¬¸» »®®±® ¾±«²¼ º±® ¬¸» ÑÔÍ øÑ®¼·²¿® § Ô»¿¬ Í¯«¿®»÷ ±´«¬·±² ¿ ©»´´¬¸» ÓÔÛ ±´«¬·±²ò Ý±®±´´¿® § ì øÎ»«´¬ º±® Ð±±´»¼ Ó±¼»´ô Ù¿«·¿² Ü¿¬¿ô ×²¼»°»²¼»²¬ Î±©÷ò Ô»¬ È ¾» ¿³°´»¼ ò¬ò
Ô±©»® Þ±«²¼ º±® ÑÔÍ ¿²¼ ÓÔÛae É» ²±© ¸±© ¬¸¿¬ ¬¸» »®®±® ¾±«²¼ º±® ¾±¬¸¬¸» ÑÔÍ ¿ ©»´´¿ ¬¸» ÓÔÛ ±´«¬·±² ¬¿¬»¼ ¿¾±ª» ¿®» ·² º¿½¬ ¬·¹¸¬ «°¬±´±¹óº¿½¬±®ò Ô»³³¿ ëò Ô»¬ ¬¸» ¿«³°¬·±² ±º Ý±®±´´¿® § ì¸±´¼ò Ì¸»²ô ©»¸¿ª» ø©ò°ò ï ïã²
Ë·²¹ Ì¸»±®»³ î ©·¬¸Ì¸»±®»³ ïë ¿²¼ øíî÷ ø©·¬¸½»®¬¿·² ßô Þ÷ ©» ±¾¬¿·² ¬¸» º±´´±©·²¹ ½±®±´´¿® §ò Ý±®±´´¿® § ê øÎ»«´¬ º±® Ð±±´»¼ Ó±¼»´ô Ù¿«·¿² Ü¿¬¿ô Ü»°»²¼»²¬ Î±©÷ò Ô»¬ È ¿¾±ª»ò Ô»¬ ² Ýø³ õ ¼÷´±¹ø³ õ ¼÷ò Ì¸»² ¬¸» º±´´±©·²¹¸±´¼ ø©ò°ò ï Ì ã² ïð ÷ae
Ø»®» ¿¹¿·²ô ¬¸» ÓÔÛ°®±¾´»³ · ²±²ó½±²ª»¨¿²¼¸»²½» ¬¿²¼¿®¼ ¿²¿´ §· ¼±» ²±¬°®±ª·¼» ¬®±²¹ ½±²ª»®¹»²½» ¹«¿®¿²¬»»ò Í¬·´´ô ¿´¬»®²¿¬·²¹ ³·²·³·¦¿¬·±² ø±º ²»¹¿¬·ª»´±¹ó´·µ»´·¸±±¼÷ º±® É å´»¿¼ ê ß´¹±®·¬¸³ î ß´¬Ó·²óÍËÎae ß´¬»®²¿¬·²¹ Ó·²·³·¦¿¬·±² º±® ÍËÎ Î»¯«·®»ae Ü ã ºøÈ ï å § ï ÷ ae ae ae øÈ î å § î ÷¹ô Ò«³¾»® ±º ·¬»®¿¬·±²ae Ì ïae Î¿²¼±³´ §°¿®¬·¬·±² Ü ã ºÜ ð å Ü ð å Ü ï å Ü ï å ae ae ae å Ü å Ü ¹ô ©¸»®» ¶Ü © ¶ ã ¶Ü ¶ ã ²ô è¬ îae ×²·¬·¿´·¦» É ð ã ð íae º±® ¬ ã ðå ae ae ae å Ì ï ¼±

SUMMARY:This paper studies the effect of noise correlation in some models of multi-output regression. It argues that a method that does not benefit from the correlation, such as Ordinary Least Squares (OLS), may perform much worse than a method that does, such as Maximum Likelihood Estimation (MLE). For certain linear models (Pooled model and Seemingly Unrelated Regression), which are studied in the paper, the MLE estimator requires the joint optimization of the covariance and regression weights. This is a non-convex problem. Alternative Minimization (AltMin) algorithm is an approach to solve the problem by iteratively optimizing the covariance and the weights. This is not guaranteed to find the global optimum.The contribution of the paper is a finite-sample study of AltMin and MLE and showing the effect of correlation. The results indicate that both MLE and AltMin can in fact benefit from the correlation, but OLS cannot. For certain examples, the difference can be significant. The paper provides lower bound for OLS and MLE (for the Pooled model) that up to a logarithmic factor in the number of samples is the same as the upper bound. This indicates that the upper bound of AltMin, which is the same as MLE's, is indeed tight.The paper has some empirical studies too. EVALUATION:I believe this is a reasonably good paper. It provides intuition, has theory, and performs simple experiments that validate the theory. I have a few comments:- The eps in the experiments is very small, indicating highly correlated outputs. Would the difference between MLE/AltMin and OLS be still significant when eps is much larger (maybe in the order of 0.1-0.5)? The theory suggests that the behaviour is O(eps), but what about the effect of other constants, which might be different between OLS and AltMin/MLE?- The paper assumes that the noise model is Gaussian. How crucial is this for the conclusions of the paper? It appears that the only step that requires this Gaussian assumption is Lemma 11, which is Corollary 5.35 of [28]. It seems that one can safely use results of Section 5.3.2 of [28] to extend the results to beyond Gaussian noise models. Is this correct?- Some typos in the proof of Theorem 15 (p13):+ L650: The summation over i is missing in the RHS. + L656: zeta_1 is on top of the inequality symbol, but is never referred to. + L663: It seems that a log n term is missing after the equality. - The proof of Lemma 3, L710: The inequality indicated by zeta_1 holds with a probability at least 1 - exp(-cn), but it is not stated and taken care of. ===== UPDATE: Thank you. You addressed my concerns.


Mind the Gap: A Generative Approach to Interpretable Feature Selection and Extraction
We present the Mind the Gap Model (MGM), an approach for interpretable feature extraction and selection. By placing interpretability criteria directly into the model, we allow for the model to both optimize parameters related to interpretability and to directly report a global set of distinguishable dimensions to assist with further data exploration and hypothesis generation. MGM extracts distinguishing features on real-world datasets of animal features, recipes ingredients, and disease co-occurrence. It also maintains or improves performance when compared to related approaches. We perform a user study with domain experts to show the MGM's ability to help with dataset exploration.

(Light Review) This paper presents a clustering method that     purports to provide greater interpretability for what     distinguishes one cluster from another.The authors do this by     first providing a feature selection method that creates logical     rules to group sets of features that vary together, and second     by modeling distinguishing parameters as having a bi-modal     distribution (i.e., with a gap in feature values that     distinguishes different clusters) vs. non-distinguishing     parameters as having a unimodal distribution (possibly of high     variance).These choices are well motivated and the development     of the technique is relatively clear; the work also appears to     be novel in its approach.Results are shown first in terms of     mutual information with underlying labeled clusters, for which     it performs in the top two for the set of baselines presented in     both data scenarios.However, the main point of the technique     (and the paper) was greater interpretabilty, and the evaluation     for this is very weak - the authors essentially talk about the     features that were found as distinguishing and explain that     they're reasonable.While their explanations are sensible,     they're ultimately insufficient to justify the premise of the     paper, as (a) such a subjective judgment requires multiple     judgments by human subjects (via Turk, etc.) rather than     arguments by the authors, and (b) no baseline for comparison is     shown, while in fact there are other methods that purport to     describe the differences between clusters (albeit focused on the     most discriminating features, as the authors point out).Since     this was presented as the primary contribution of the paper, it     seems that the authors have not sufficiently demonstrated that     their method has satisfied this claim.