The Bayesian Case Model: A Generative Approach for Case-Based Reasoning and Prototype Classification
We present the Bayesian Case Model (BCM), a general framework for Bayesian case-based reasoning (CBR) and prototype classification and clustering. BCM brings the intuitive power of CBR to a Bayesian generative framework. The BCM learns prototypes, the "quintessential" observations that best represent clusters in a dataset, by performing joint inference on cluster labels, prototypes and important features. Simultaneously, BCM pursues sparsity by learning subspaces, the sets of features that play important roles in the characterization of the prototypes. The prototype and subspace representation provides quantitative benefits in interpretability while preserving classification accuracy. Human subject experiments verify statistically significant improvements to participants' understanding when using explanations produced by BCM, compared to those given by prior art.

This paper proposes the Latent Case Model (LCM), a Bayesian      approach to clustering in which clusters are represented by a      prototype (a specific sample from the data) and feature      subspaces (a binary subset of the variables signifying those      features that are relevant to the class).  The approach is      presented as being a Bayesian, trainable version of the      Case-Based Reasoning approach popular in AI, and is motivated by      the ways such models have proved highly effective in explaining      human decision making.  The generative model (Figure 1)      represents each item as coming from a mixture of S clusters,      where each cluster is represented by a prototype and subspace      (as above) and a function \phi which generates features matching      those of the prototype with high probability for features in the      subspace, and uniform features outside it. The model is thus      similar in functionality to LDA but quite different in terms of      its representation. Experimental results are shown for      "prediction accuracy" (more on this later) and unsupervised      accuracy of the clusters vs. a baseline of LDA (Figure 2), and      the results seem superior/comparable to LDA.  Further      experiments are shown regarding the interpretability of the      models - human subjects are asked to categorize prototypes from      LCM vs. lists of top features from LDA; the LCM condition      significantly outperformed LDA (85.9% vs. 71.3%).      I greatly enjoyed this paper - not only does it present a novel      method that truly stands out in the vast sea of conventional      LDA-like models, it is refreshingly well-written - the      motivation and background were compelling, the description of      the model and its elements (Section 3) was frankly among the      best I've ever seen, the motivating example (section 3.1) was      clear and extremely helpful, and the experiments were      illustrative (though lacked clarity in places, more details      below). Interpretable models are extremely important, especially      for unsupervised tasks such as clustering - I expect that many      in the NIPS community will find this work compelling and may      want to try this model or variants in their own work.      That said, I do have a few issues, and they have mostly to do      with clarity in the experiments section.  First, in Figure 2a,      I'm a bit confused as to what is going on - the authors are      using unsupervised models on digit/newsgroup data, and then      according to the text are training an SVM based on the labels      from those clusters?  To what end? And then what is that SVM      being used to classify - some held out data?  This is really      unclear, at least to me - the authors do cite [6], which likely      explains it, but really the experimental procedure here should      be fully explained.      The second set of issues have to do with the user study.  While      I applaud the authors for having done this study and properly      randomizing orders, conditions, etc., I had a minor concern.  On      line 361, when describing the target categories for the task,      the authors say "For LCM, we simply 'presented the prototype'" -      what does that mean - were subjects shown the recipe in its      entirety (all ingredients) or just the *name* of the recipe      (i.e., "fudge brownie")?  This needs to be clarified in the      text.  Also, if it is the latter (as I suspect, as in Figure 4      the 'Prototype' column is the recipe name), it seems somewhat      unfair to show only a list of top features (ingredients) for the      baseline (LDA) condition - instead they could show a "prototype"      for each cluster there as well by simply selecting the item      that had the highest posterior for that class (this is a common      heuristic to get a representative sample from an LDA topic).      This is not a huge deal, and likely there wouldn't be time to      run the study again, but this should at least be mentioned in      the discussion.      In any case, the issues above are quite minor; I think this is a      significant and well-written paper that would get a good deal of      positive attention at this year's conference.


A Dual Algorithm for Olfactory Computation in the Locust Brain
We study the early locust olfactory system in an attempt to explain its wellcharacterized structure and dynamics. We first propose its computational function as recovery of high-dimensional sparse olfactory signals from a small number of measurements. Detailed experimental knowledge about this system rules out standard algorithmic solutions to this problem. Instead, we show that solving a dual formulation of the corresponding optimisation problem yields structure and dynamics in good agreement with biological data. Further biological constraints lead us to a reduced form of this dual formulation in which the system uses independent component analysis to continuously adapt to its olfactory environment to allow accurate sparse recovery. Our work demonstrates the challenges and rewards of attempting detailed understanding of experimentally well-characterized systems.

NOTE: Due to the range in reviews, we were asked to discuss this paper. As a result of that discussion, we'd like the authors to address the three following points:1) How are the firing rates (or some other physiological measure of activity) mapped to the variables y and in particular lambda. Lambda - in the current formulation - has no bounds. How does this affect the interpretation. 2) There is a need for "negative reciprocal" connectivity between the PNs and LNs (this comes out of the use of B and B_transpose) in Equation 12. Similarly there is a need for the same symmetry between the measurements of the sparse vector x and the projections up to the KCs. Is there evidence of this symmetry in the locus anatomy or physiology? Can the model work without it? 3) So far the main evidence that this model describes the olfactory circuit in locust is that the odor input is sparse in terms of molecules per input, somewhere along the way the sparse signal is measured by the A matrix such that it is projected into a lower-dimensional, denser representation. (It should be clarified where this happens... we believe it happens on the level of the ORs such that each OR responds to several molecules and each molecule binds with several ORs). Then there is some processing in this lower-dimension via the B matrix (PNs and LNs) which is then projected back up into a higher-dimension through A to the KCs. We feel that while there is some structure of the biological circuit in the model, this paper needs to provide more evidence. Is there evidence of gradient descent dynamics? Is there evidence in the plasticity/processing between LNs and PNs that can be mapped to the form of ICA presented here? THE TEXT BELOW IS MY ORIGINAL REVIEW:Firstly, I love seeing a theoretical paper which attempts to model a very particular circuit. Typically, we see attempts to create general neural networks or to formally characterize systems which are not well characterized from a biological/experimental point of view. Here we have the opposite, which seems like a more fruitful approach, that is if one's goal is to understand nervous systems.In this paper, the authors derive a mathematical model of processing in the locust olfactory system. The input is a bit vector representing an odor where each entry encodes the presence or absence of particular molecules. Because many ORNs each with the same OR converge on a single glomeruli, the input can be taken as noise-free. In the low-dimensional glomerular layer, the mixed signals from the ORNs interact with each other via local interneurons and PNs. Finally, the PNs then project up into the higher-dimensional space of KCs where each KC codes for a particular molecule in the input.The authors first describe the problem of recovering the sparse, high-dimensional input via the KCs as a MAP optimization, but then reformulate the problem in the compressed sensing context. Next, the dual of the compressed sensing problem is presented as a model since it can more easily be interpreted in terms of the locust nervous system. This model is then changed again to further take knowledge about the biological circuit into account by ensuring that any dynamics happen on the M-dimensional signal. This requires the circuit to learn an M-dimensional subspace of the full N-dimensional space to operate in which is implemented using ICA.This paper is highly original in the way it brings together the compressed sensing problem, ICA, and a particular neural circuit. Also, the presentation of the model is clear and easy to follow. However, I would like a clearer explanation of the biological circuit and how it maps back to the model. Is it that each OR can bind with several molecules such that the glomeruli each represent a mixture of molecules? Then the PNs readout the glomeruli and also take part in the dynamics which ultimately allows the readout by the KCs which sample a mixture of PNs? The basic question is where exactly does the mixing happen. After reading this paper and reviewing references, I'm still not completely certain. I think where I get lost is whether each OR binds with several or only one molecule. It would be helpful to really nail down how the model and biology map to each other since this paper's strongest aspect is that it models a particular circuit. Maybe to make room you can condense the sections where you present models that are ultimately not used.As it stands, the current model is very interesting and I believe that with some clarifiations it is motivated by the biology. However, the real key to making this a significant paper would be to show that the dynamics of the model (either the gradient descent dynamics or something about the ICA dynamics) match the dynamics of the true circuit. So far you've shown that this model can perform sparse regression/compressed sensing and that it has some structural similarities to the biological circuit. I'd like to see a figure that compares a simulation of this model with the true circuit even if it's only that individual neuron's responses match physiological data. Or maybe there is evidence of an ICA implementation in the locust olfactory system? Do they dynamics of ICA seem comparable to any behavioral measurements from locust? Or maybe the ability to discriminate odors wouldl show that performance was more like the reduced dual than the full dual circuit? I would love to see something like this addressed in a revised version of this paper or in a future paper.


Two-Layer Feature Reduction for Sparse-Group Lasso via Decomposition of Convex Sets
Sparse-Group Lasso (SGL) has been shown to be a powerful regression technique for simultaneously discovering group and within-group sparse patterns by using a combination of the 1 and 2 norms. However, in large-scale applications, the complexity of the regularizers entails great computational challenges. In this paper, we propose a novel two-layer feature reduction method (TLFre) for SGL via a decomposition of its dual feasible set. The two-layer reduction is able to quickly identify the inactive groups and the inactive features, respectively, which are guaranteed to be absent from the sparse representation and can be removed from the optimization. Existing feature reduction methods are only applicable for sparse models with one sparsity-inducing regularizer. To our best knowledge, TLFre is the first one that is capable of dealing with multiple sparsity-inducing regularizers. Moreover, TLFre has a very low computational cost and can be integrated with any existing solvers. Experiments on both synthetic and real data sets show that TLFre improves the efficiency of SGL by orders of magnitude.

--- Summary ---This paper generalizes the screening procedure  to  the sparse group lasso problem where it is able to identify  inactive groups and inactive features (i.e., features with zero weight in the optimal solution) simultaneously before optimization and consequently to accomplish dimensionality reduction. Therefore, when used as a per-processing step in sparse group lasso problem to reduce its scale,  the computational complexity of the optimization problem can be reduced significantly without sacrificing the accuracy of the model. --- Contributions and overall evaluation ---To apply the typical screening methods, one needs to  estimate the optimal dual solution  of the original optimization problem. For sparse group lasso, the feasible set is an intersection of ellipsoids and it turns out that the dual estimation is a difficult task.  The basic idea  to remedy this difficulty is to utilize Fenchel's duality to derive an elegant decomposition of the dual feasible set (i.e., sum of simple convex sets due to the pretense of multiple sparsity inducing regularizes) to be able to estimate the dual objective. The paper does survey related work appropriately and is clearly written.  The paper technically sound and the proofs seem to be correct.  It is clear, of high quality, novel, definitely of significance.  However, more effort can be put into making the proofs more understandable (though many of the explanations are very good) and extending the experiments. -- Minor comments --- - Line 181, Page 4, by --> By- Line 527, Page 10, |\beta|_1 --> |\beta_g|_1


Clamping Variables and Approximate Inference
It was recently proved using graph covers (Ruozzi, 2012) that the Bethe partition function is upper bounded by the true partition function for a binary pairwise model that is attractive. Here we provide a new, arguably simpler proof from first principles. We make use of the idea of clamping a variable to a particular value. For an attractive model, we show that summing over the Bethe partition functions for each sub-model obtained after clamping any variable can only raise (and hence improve) the approximation. In fact, we derive a stronger result that may have other useful implications. Repeatedly clamping until we obtain a model with no cycles, where the Bethe approximation is exact, yields the result. We also provide a related lower bound on a broad class of approximate partition functions of general pairwise multi-label models that depends only on the topology. We demonstrate that clamping a few wisely chosen variables can be of practical value by dramatically reducing approximation error.

Review for On the Power of ClampingSummary: This paper presents some deep technical results on inferencein a particular class of undirected graphical models.  By clampingwell-chosen variables, a complex graph can be split into smallergraphs, which can each be processed independently, and the resultscombined.  The authors present approximation bounds for the partitionfunction in such an approach, and demonstrate that clamping improvesapproximation error.This paper is a well-written deep dive into a specific inferenceapproach on a specific model class (binary pairwise model withattractive potentials).  Proofs of the main technical results are thecenterpiece of the paper, although some simple experiments demonstratethat the approach can have practical value by reducing approximationerror.Overall I found this paper well-done, although a bit dense at times.While it's unclear to me how generally interesting these results are(given the fairly limited class of applicable problems), the materialpushes the state-of-the-art in a long line of similar work that thecommunity considers valuable.One weakness of this paper is the general lack of connection to otherapproximation methods.  As the authors note, the idea of conditioningon a variable (or set of variables) and approximating over the resthas a long and fruitful history.  Given that rich and variedliterature, it would have been interesting to see how other methodscompare to clamping+Bethe, either theoretically or practically. Theauthors do compare to TRW in the experiments, but I'm left wonderingif there are other serious approximation methods for the same class ofmodels, and if so, what their performance/guarantees might look like.Along those lines, I strongly encourage the authors to pursue the ideaoutlined in the last paragraph: "It is tempting to speculate thatsimilar results may be found for other approximations. For example,for some methods that upper bound the partition function, such as TRW,it may be possible to show that a lower (hence better) approximationis always achieved when a variable is clamped."  That would beinteresting indeed.Minor comments:You prove that Z_B <= Z, and that Z_B^i >= Z_B.  Perhaps I missed it,but can you say anything about the relationship between Z_B^i and Z?Is clamping upper-bounded by Z?


Multitask learning meets tensor factorization: task imputation via convex optimization
We study a multitask learning problem in which each task is parametrized by a weight vector and indexed by a pair of indices, which can be e.g, (consumer, time). The weight vectors can be collected into a tensor and the (multilinear-)rank of the tensor controls the amount of sharing of information among tasks. Two types of convex relaxations have recently been proposed for the tensor multilinear rank. However, we argue that both of them are not optimal in the context of multitask learning in which the dimensions or multilinear rank are typically heterogeneous. We propose a new norm, which we call the scaled latent trace norm and analyze the excess risk of all the three norms. The results apply to various settings including matrix and tensor completion, multitask learning, and multilinear multitask learning. Both the theory and experiments support the advantage of the new norm when the tensor is not equal-sized and we do not a priori know which mode is low rank.

After reading the authors feedback, I think I have misunderstood the "mode" in the experiment, which is exactly what I was requested. I have modified my score correspondingly. ===In this paper the authors studied an approach for multi-task learning, which represent the (linear) models of the tasks into a tensor, and enforce a low-rank structure in the tensor to model the task relatedness, in which the knowledge is transferred among the tasks. The existing approaches such as latent trace norm failed to address the problem where the task dimensions are heterogeneous, and this paper proposed an improve version of latent trace norm, which normalizes along each dimension. The authors derived error bounds for the two existing tensor norms and the proposed one, relating them to the expected dual norm and showing the expected dual norm for the three norms. It is really nice to have a unified comparison over different norms as shown in the paper. Finally, the authors showed experimental results on synthetic data as well as two real-world data. Overall, this is a good paper with interesting theoretical analysis, and below are my comments of the paper. My main concern of this paper is the usefulness of the tensor multi-task learning. Indeed we can have some “features X aspects X customers” model to explore the relatedness, and the question is that: is it necessary to go beyond two-dimensional case. For any tensor model, we can collapse the model into a flat “features X task” matrix and enforce a low rank to enforce the task relatedness. As one may argue this cannot capture some part of the information as did in tensor formulations, it rare to see how tensor can indeed increase the performance. Even if it helps a little bit, will people pay a large amount of additional time cost for such increase in performance? The lacking of the comparison to simple matrix trace norm in the experiments somehow exaggerated my concern. The core novelty of this paper is the scaled version of a traditional latent trace norm, by adding a scaling factor to the unfolding along each mode. This suggests that we should treat each task (and the unfolding) differently due to some reasons. The idea is somehow related to another paper trying to address a similar problem, the “Multivariate Regression with Calibration” by Liu et. al. The authors may want to see if this paper can be related to that one.  It is interesting if the authors can show some connection between the two. What is the algorithm used in this paper to solve the formulation with the new norm? Is it the same as the one used in the original latent tensor norm? What is the complexity of algorithm? In the case where scaled latent trace norm performs better than the original version, how much time does it take to converge? As mentioned before, for multi-task learning, everything use tensor can also be used by flat trace norm. The authors should at least compare to these types of “flat” multi-task learning methods, e.g., the trace norm, in terms of performance and efficiency.  Also, it is weird to see that the authors use different evaluation (explained variance) metric for the school data. There are many multi-task learning papers that report MSE/NMSE for school data. The authors can use MSE on school data as well to make the paper consistent.Some minor comments:1. Notation “W_{(k)}^{(k)}” in (2) and later on is very confusing. It has not been defined before its use. 2. Typo “Turing”=>“Turning” in page 5, 10 lines from the bottom.


A Latent Source Model for Online Collaborative Filtering
Despite the prevalence of collaborative filtering in recommendation systems, there has been little theoretical development on why and how well it works, especially in the "online" setting, where items are recommended to users over time. We address this theoretical gap by introducing a model for online recommendation systems, cast item recommendation under the model as a learning problem, and analyze the performance of a cosine-similarity collaborative filtering method. In our model, each of n users either likes or dislikes each of m items. We assume there to be k types of users, and all the users of a given type share a common string of probabilities determining the chance of liking each item. At each time step, we recommend an item to each user, where a key distinction from related bandit literature is that once a user consumes an item (e.g., watches a movie), then that item cannot be recommended to the same user again. The goal is to maximize the number of likable items recommended to users over time. Our main result establishes that after nearly log(km) initial learning time steps, a simple collaborative filtering algorithm achieves essentially optimal performance without knowing k. The algorithm has an exploitation step that uses cosine similarity and two types of exploration steps, one to explore the space of items (standard in the literature) and the other to explore similarity between users (novel to this work).

** COMMENTS AFTER AUTHOR FEEDBACK **This is a high quality paper.Minor comment. Not addressed in the rebuttal is"1. The definition of neighbourhood is very unclear, because of the use of “jointly”. Line 178 ... contradicts Lemma 8’s proof (Line 576)."which might be good to clarify in the paper.--------------------------------------------------Disclaimer: the reviewer is not an expert in Learning Theory, but in Collaborative Filtering. In assessing the paper’s theoretical significance, he/she will submit to the viewpoint of expert Learning Theorists. MAIN IDEAS OF THE PAPER:We know that collaborative filtering works in practice: users who buy A also tend to buy B. The paper takes a bold stab at providing theoretical guarantees on how well it works, albeit in a restricted and slightly hypothetical setting. The hypothetical setting is a simplified sketch of reality, necessary to obtain a theoretically analysable handle on the problem.The paper analyses a very simple collaborative filtering algorithm, where collaborative filtering relies on looking for a set of neighbouring users with the most similar like/dislike patters (within some cosine distance) and using their extra like/dislike votes to score items to provide next item to a user. The goal is to maximize the expected number of liked items presented to each user within a fixed time window of 1 to T recommendations.The paper shows that the users are “learnable” after around log(km) = log(number of user types) + log(number of items) rounds of recommendations (up to log^{7/3}(km), depending on a parameter choice). Essentially in time logarithmic in the number of items, roughly optimal performance can be expected. The whole set-up relies on some conditions, like having enough users.The result relies on two additional steps in the collaborative filtering algorithm: (1) random exploration, and (2) structured exploration. Both are necessary, (1) to interrogate potential future items for recommendation, and (2) to have enough of an overlap in like/dislike items to learn about the similarity between users.The experimental results try to simulate a real scenario by streaming a curated subset of the Netflix prize data to the data set’s users. In the simulation, the average cumulative reward is higher than those of two existing methods (PAF and DM, incidentally neither of which has ever been used in practice, to my knowledge and those of peers).RELATIONSHIP TO PREVIOUS WORK:The paper is set apart by using two exploration steps, one of which learns the similarity between users (setting it apart from standard multi-arm bandits). Furthermore, no arm can be pulled more than once.QUALITY and CLARITY:The paper is very well explained, and beautifully written.I worked through most of the proofs in the supplementary material, and they seem technically correct.I would like to comment on some textual issues for the benefit of the authors:1.  The definition of neighbourhood is very unclear, because of the use of “jointly”. Line 178 suggests that *only* user-item indexes that were recommended in the joint exploration step in the algorithm are eligible for defining neighbours (and hence cosine similarity). Why are other overlapping items not considered in defining the neighbourhood? The confusion arises as Line 178 is intentional, but if we read Lemma 8’s proof (Line 576), it contradicts Line 178, as now jointly means the items that two users rated in common, and *not* the subset of that set, which was tagged with “jointly” in your Algorithm (see Line 157).2. Sec 2, Lines 103 and others. The paper exclaims the point that one should not recommend a consumed item to a user again (especially a disliked one). This is rather obvious in the recommendations community, and I assume you drive home the point to distinguish your work from a standard multi-arm bandit setting?3. Typos.Line 165 should refer to \tilde{p}_{ui} and not index j?Line 524 (Lemma 5’s proof). \bar{Z}_s = 1 / s^\alpha – Z_s would give a zero mean.Line 610 [unnecessary inequality in proof]Line 634 user u has is...ORIGINALITY and SIGNIFICANCE:This paper is the first attempt (to my knowledge) to characterize learning rates in a (pseudo-) real recommendations setting.The (new) practical take-home message is that joint exploration is required for the algorithm (and maybe more generally?) to achieve optimality in time logarithmic in the number of items.PRACTICAL WEAKNESSES TO BE AWARE OF:To achieve the main results (that is, how long and how much should one explore and exploit before attaining optimal performance) the proofs and algorithm rely on a number of assumptions. These assumptions already give useful insights (like how many users one needs relative to an item catalogue size and tolerance), but from a practitioner’s viewpoint, they break down on many levels.1. The assumption that users belong to each type with odds 1/k is invalid. In reality, these are typically according some power law.2. Random (and joint) exploration, whilst theoretically appealing (and necessary) is a dangerous game, as it can hurt user retention in systems like those described in the Introduction. The algorithm doesn’t make provision for a user never returning because of it.3. From calibration plots in real systems, we know that condition A1 does not hold in practice. There are pairs for which the odds are really, well, coinflip!These are not criticisms, but differences to be aware of.


Robust Tensor Decomposition with Gross Corruption
In this paper, we study the statistical performance of robust tensor decomposition with gross corruption. The observations are noisy realization of the superposition of a low-rank tensor W * and an entrywise sparse corruption tensor V * . Unlike conventional noise with bounded variance in previous convex tensor decomposition analysis, the magnitude of the gross corruption can be arbitrary large. We show that under certain conditions, the true low-rank tensor as well as the sparse corruption tensor can be recovered simultaneously. Our theory yields nonasymptotic Frobenius-norm estimation error bounds for each tensor separately. We show through numerical experiments that our theory can precisely predict the scaling behavior in practice.

This paper studies low rank tensor recovery problem with gross corruption using convex program. The convex programs considered here seem to be inspired by matrix completion and tensor trace norms. The main contribution is in handling gross corruption, which is a very natural noise model. The paper also involves some synthetic experiments.Overall this is an interesting paper. The analysis is clean. The results are reasonable but the bounds involves many parameters and would be good if the authors can give more intuitions on different trade-offs. For example, although Assumption 1 is reasonable (e.g. it holds for random subspace \Omega and random sparsity pattern S as long as the rank/sparsity is not too large), it is not immediately clear and would be good to discuss some typical cases where it is true. It would also be good if the bound can be simplified (for example in the case where all the n_k's are the same, r_k's are also the same) to provide more intuition.Another (slight) concern is that the authors are mixing tensor decomposition with tensor completion in the intro. Although these are closely related, tensor decomposition (especially CP decomposition) is often harder than tensor completion (also for low rank tensor completion typically the definition of rank is different from CP rank, as in this paper).


Nonparametric Bayesian inference on multivariate exponential families
We develop a model by choosing the maximum entropy distribution from the set of models satisfying certain smoothness and independence criteria; we show that inference on this model generalizes local kernel estimation to the context of Bayesian inference on stochastic processes. Our model enables Bayesian inference in contexts when standard techniques like Gaussian process inference are too expensive to apply. Exact inference on our model is possible for any likelihood function from the exponential family. Inference is then highly efficient, requiring only O (log N ) time and O (N ) space at run time. We demonstrate our algorithm on several problems and show quantifiable improvement in both speed and performance relative to models based on the Gaussian process.

Brief Discussion:The paper presents a method for generalisation of local kernel methods to the Bayesian stochastic process setting. In particular they define a smoothness condition that allows them to obtain under certain assumptions a link between a Guassian process kernel and the smoothness condition imposed on the likelihood.QualityThe paper is well written but would benefit from more detailed discussion on the relationship between the kernel and the function rho(.,.) for smoothness. In particular some examples and discussion on whether this is always possible to achieve for any class of kernel would be beneficial. This could be done directly after Theorem 1. In addition, the results presented on the extended likelihood could be more carefully explained and discussed.The authors should further motivate the assumption of independence which does hold in general settings - what are the settings this is reasonable in practice, what are the implications of this assumption on the inference otherwise in practical examples proposed. Explain further how it relates to the other methods cited in this section on sparsification - what exactly is the relationships perhaps make a small illustrative example of the link. In addition the discussion on the equivalence of the interaction between input variables either in the likelihood function or in the prior is interesting and should be expanded. In practical settings which would you choose in general to model the interaction in the likelihood or the prior, comment on this remark, and is this equivalence always true - under what conditions? please comment.Clarity - the paper is generally well written and conciseOriginality - the paper is interesting though the contributions are incremental in natureSignficance - the paper is interesting for the audience.


Neural Word Embedding as Implicit Matrix Factorization
We analyze skip-gram with negative-sampling (SGNS), a word embedding method introduced by Mikolov et al., and show that it is implicitly factorizing a word-context matrix, whose cells are the pointwise mutual information (PMI) of the respective word and context pairs, shifted by a global constant. We find that another embedding method, NCE, is implicitly factorizing a similar matrix, where each cell is the (shifted) log conditional probability of a word given its context. We show that using a sparse Shifted Positive PMI word-context matrix to represent words improves results on two word similarity tasks and one of two analogy tasks. When dense low-dimensional vectors are preferred, exact factorization with SVD can achieve solutions that are at least as good as SGNS's solutions for word similarity tasks. On analogy questions SGNS remains superior to SVD. We conjecture that this stems from the weighted nature of SGNS's factorization.

The paper investigates the skip-gram model of Mikolov et al., more specifically, the more successful 'negative sampling' version.  The submission draws a connection between the word2vec objective and a form of PMI (pointwise mutual information) measure, and, guided by these analyses, proposes a simple modification of PMI (shifting it by a constant, and replacing negative values with zero). The paper is generally well-written and interesting in many aspects, however I have a few reservations:1)  The connection between the skip gram model and factorizing log P(w | c ) / P(w) is  unsurprising, especially for the soft-max version of the model (also recall the fairly established practice of applying LSA in log space). Still, it is nice to see a more formal look into this connection for the negative-sampling version.2) The authors may miss another role  of the constant k in the context of word2vec (and perhaps more important one?). Unlike, their theoretical analyses, word2vec does not perform marginalization over the entire vocabulary and multiplication of this expectation by k, but rather draws k samples. Thus, another role of k is to produce a more reasonable approximation of the corresponding expectation (~ the softmax partition function). This may be especially important given that word2vec is normally run only for a very small number of iterations over the training set.  It may be interesting to see if what matters is the 'shift' role of k (as argued by the authors) or the approximation / exploration role of k.  This is actually fairly easy to do by introducing 2 constants (m and l) instead of k. Instead of considering k negative samples and summing over them,  one could consider m samples, normalize them (i.e. divide the sum by m) and multiply the result by l.  This would help to disentangle the 2 roles: increasing m would improve the quality of the approximation (without affecting the shift), and increasing l would increase the 'shift' (without affecting the quality of the approximation).  3) Overall, this "shifting-by-log k" extension seems a little ad-hoc (and not so helpful in practice). I am not sure I can really buy its motivation — it seems to be motivated by the fact that the negative sampling objective is ‘using’ the ‘shift’. But it is not really clear that this is really desirable, and it is not clear whether this is really that useful (see the point (2) above), especially in the context of PMI.  On the one hand it results in ignoring low ‘correlations’ but on the other it is doing more than that as the cosine similarity is not shift invariant.  Some more discussion / analyses would be helpful.24 The authors look only into the local objective (i.e. partial derivative of the objective on a specific word-context pair). It may make more sense to look into the global objective, as this objective explicitly incorporates weighting for the individual terms in the factorization problem.  This is mentioned in section 3.2 but I was surprised that it has not been done in 3.1.5) I am really not sure what these small values for SPPMI in Table 1 are telling to us.  SPPMI is different from the objective only for negative terms, so this is probably where the difference is coming from. Is this delta so small because negative terms are infrequent? I am slightly unsure if the results are also affected by the quality of the objective approximation you are using (here I refer to the approximations used to compute the objective, as described in section 5.1). Some additional discussion may be helpful. 6) Overall, section 5.1 is somewhat questionable as SPPMI does not really optimize the objective as no embeddings are really induced. For any objective function, optimizing terms independently (i.e. ignoring any interaction between them) would result in a better 'solution'.  Not really sure what this tells us though. 7) Though it is interesting to see that using PPMI results in comparable or better results on many tasks, this is not a new result as  similar observations were made in Levy and Goldberg (CoNLL 2014, cited).


Incremental Clustering: The Case for Extra Clusters
The explosion in the amount of data available for analysis often necessitates a transition from batch to incremental clustering methods, which process one element at a time and typically store only a small subset of the data. In this paper, we initiate the formal analysis of incremental clustering methods focusing on the types of cluster structure that they are able to detect. We find that the incremental setting is strictly weaker than the batch model, proving that a fundamental class of cluster structures that can readily be detected in the batch setting is impossible to identify using any incremental method. Furthermore, we show how the limitations of incremental clustering can be overcome by allowing additional clusters.

This paper presents a number of theoretical results characterizing the limitations/capabilities of incremental clustering. Below I summarize some of the key results, skipping some of less significance (from my point of view). One of the key results is that there exists no deterministic memory-bounded incremental method that can guarantee to detect nice clustering (i.e., a clustering of data in which every element is closer to points in its own cluster than to points in other clusters) from data when a unique nice clustering exists. The main idea of the proof  is to construct a data set with a unique nice clustering and demonstrat that any deterministic incremental algorithm that's memory bounded will necessarily fail with some ordering of the data. The paper also shows that perfect clustering, a stronger notion of clusterability, can be detected by a single-link based incremental algorithm, but not by the popular incremental k-means. By allowing extra clusters, the paper shows that an incremental algorithm can find a refinement of the target nice clustering if allowed 2^{k-1} centers and also shows that it is not possible to use fewer cluster centers.Given that the definition of nice clustering requires a somewhat global view of the clustering structure, it is somewhat intuitive that memory bounded incremental algorithms will fail. Non-the-less, the lower bound results are quite interesting and original (to the best of my knowledge). I followed most of the proofs provided in the paper as well as the appendix and they appear to be sound and some of them non-trivial.  I think they present solid contribution to the theoretical understanding of incremental clustering.  The presentation of the paper has some room for improvement. In particular, the paper itself lacks balance in its space allocation. The first result took 6 pages and was given the (almost) full proof, whereas the later results are provided without any proof or even sketch of proof.  A reader will have to read the supplementary materials to get through the rest of the paper. Further more, the supplementary materials were not separated from the main paper. Rather, an extended version was presented as the supplementary. This makes it difficult to find the proof for a particular theorem because the numbers are not consistent.  I appreciate the attempt to provide the high level intuition for the proof of the first result, but I find the explanation difficult to understand without the detailed proof. I think this explanation needs some work to make it really useful (perhaps with some figures?).  Another issue with the presentation is that it seems to be trying to fit too much into the paper and some of the results seem dis-connected from the rest of the paper. For example, the convex-nice seems to be mainly introduced to help sequential k-means? All previous sections focus on deterministic algorithms, whereas section 5 jumps to a random algorithm, which again seems only loosely connected.Another thing on my wish list for this paper is some concrete practical understanding of these results. For example, the lower bound result on using extra clusters is quite discouraging, requiring at least 2^{k-1} clusters in order to find a proper refinement of the target nice-clustering. In practice, will this be what we typically observe? I understand that this is a theoretical paper, none-the-less, it would have been nice to see some (perhaps) synthetic experiments to demonstrate what to expect in reality when the proposed algorithm is put to use.


Inferring synaptic conductances from spike trains under a biophysically inspired point process model
A popular approach to neural characterization describes neural responses in terms of a cascade of linear and nonlinear stages: a linear filter to describe stimulus integration, followed by a nonlinear function to convert the filter output to spike rate. However, real neurons respond to stimuli in a manner that depends on the nonlinear integration of excitatory and inhibitory synaptic inputs. Here we introduce a biophysically inspired point process model that explicitly incorporates stimulus-induced changes in synaptic conductance in a dynamical model of neuronal membrane potential. Our work makes two important contributions. First, on a theoretical level, it offers a novel interpretation of the popular generalized linear model (GLM) for neural spike trains. We show that the classic GLM is a special case of our conductance-based model in which the stimulus linearly modulates excitatory and inhibitory conductances in an equal and opposite "push-pull" fashion. Our model can therefore be viewed as a direct extension of the GLM in which we relax these constraints; the resulting model can exhibit shunting as well as hyperpolarizing inhibition, and time-varying changes in both gain and membrane time constant. Second, on a practical level, we show that our model provides a tractable model of spike responses in early sensory neurons that is both more accurate and more interpretable than the GLM. Most importantly, we show that we can accurately infer intracellular synaptic conductances from extracellularly recorded spike trains. We validate these estimates using direct intracellular measurements of excitatory and inhibitory conductances in parasol retinal ganglion cells. The stimulus-dependence of both excitatory and inhibitory conductances can be well described by a linear-nonlinear cascade, with the filter driving inhibition exhibiting opposite sign and a slight delay relative to the filter driving excitation. We show that the model fit to extracellular spike trains can predict excitatory and inhibitory conductances elicited by novel stimuli with nearly the same accuracy as a model trained directly with intracellular conductances.

The authors propose a conductance based spiking model (CBSM) that is more biophysically realistic than the currently popular generalized linear model (GLM). Furthermore, the authors present CBSM as a generalization of the GLM and propose a set of constraints that can reduce it to a GLM and a GLM variant that would be as adaptive as the CBSM.The proposed model is an interesting extension to current spiking models in that it is parametrizedin a more descriptive way of the spiking process without sacrificing much of the mathematicalconvenience of the GLM. One thing that could raise some concerns stems from the last paragraph of page 6. The behaviour of the inhibitory filter described in line 322 is reminiscent of overfitting. The authors deal with thatby adding the regularizing penalty of Eq. 22. However, it seems that they don't use cross-validation in the experimental results to show the applicability of their model in a more natural environment.Some small things that might need correction:	line 121: fourth subscript should be i?	line 214: chose our an?


Online Optimization for Max-Norm Regularization
Max-norm regularizer has been extensively studied in the last decade as it promotes an effective low rank estimation of the underlying data. However, maxnorm regularized problems are typically formulated and solved in a batch manner, which prevents it from processing big data due to possible memory bottleneck. In this paper, we propose an online algorithm for solving max-norm regularized problems that is scalable to large problems. Particularly, we consider the matrix decomposition problem as an example, although our analysis can also be applied in other problems such as matrix completion. The key technique in our algorithm is to reformulate the max-norm into a matrix factorization form, consisting of a basis component and a coefficients one. In this way, we can solve the optimal basis and coefficients alternatively. We prove that the basis produced by our algorithm converges to a stationary point asymptotically. Experiments demonstrate encouraging results for the effectiveness and robustness of our algorithm.

The paper considers a problem where a matrix Z is to be decomposed into a low-max-norm matrix X, and a sparse matrix E, which is formulated as a convex problem with max-norm and L1 regularization. This can be solved using batch methods, but the focus here is on an online algorithm, which does a single pass over Z's rows. The main trick here is to reduce the original problem to another problem, which decomposes over Z's rows, and hence can be attacked using an online learning procedure. The paper proposes an algorithm, proves that it asymptotically converges to a stationary point, and provides experiments on synthetic data.I liked the basic idea of considering online matrix decomposition using the max-norm, which is an interesting and less-explored alternative to the nuclear norm. The algorithm derivation uses a simple but non-trivial trick, and the experiments seem promising. On the other hand:- I had problems with understanding the motivation to the problem through matrix completion: First, in matrix completion the data matrices are very sparse, so there is no reason not to use batch methods. Second, I couldn't follow the explanation in section 1.2 why matrix completion can be cast into this framework: I understand that the equation in line (80) is equivalent to matrix completion when C -> infinity, but why is it true that the equation is equivalent to the proposed framework (equation 1.1)? I'd appreciate it if the authors can explain this in their rebuttal.- The algorithmic derivation appears to be quite similar to OR-PCA (cited paper [8]), with the twist of using the max-norm rather than the nuclear norm, and an additional trick to get decomposability along rows. This makes this work a bit incremental.- The analysis, although far from trivial, results in a rather weak asymptotic convergence guarantee to some stationary point. This appears to be weaker than what can be obtained for related algorithms which use other approaches, such as OR-PCA. - The algorithm is computationally demanding: At each iteration, we need to solve a d*d matrix optimization problem, where d >= ambient data dimension to the best of my understanding (see detailed comments below). Although warm-start partially alleviates this, it's not clear if we end up with an algorithm whose runtime is much better than a batch method. Overall, I think the paper has some interesting ideas, but also a few significant weaknesses.Other issues/comments--------------------- Assumptions in section 4.1: It is stated that they can be made valid by adding a bit of regularization, but this also introduces additional terms into the optimization problem, so it's not clear if the result would still hold.. This should be clarified.- Is it possible to use the algorithm in a batch setting, by sub-sampling individual rows uniformly at random?- In the proposed algorithm, one needs to point out how d should be chosen. To the best of my knowledge, one needs to pick d >= max{p,n} to ensure that proposition 2.1 still holds. In that case, this issue should be explicitly discussed.Minor comments------------------- The manuscript is in A4 size rather than letter size.- line 45: "even" -> "even if"- line 56: "...E, we aim..." - comma should be period.- The introduction motivates the max-norm but formulation (1.1) also includes an additional E component with L1 regularization - this should be more clearly motivated (e.g. to handle sparse contamination)- Line 88: "in literature" -> "in the literature".- Line 173: For equation 2.7 to be true, one needs to assume that each row of the matrix Z is sampled i.i.d. from some underlying distribution. I didn't see this explicitly stated anywhere..- Line 181: "alternative" -> "alternating"(?)- LIne 232: Should ||r||_2 be ||r||_2^2? Otherwise the expression may not be strongly convex.- Line 350: "groundtruth"- Experiments: What value of d did you take? - Line 404: "In specific" -> "specifically"- The authors mention at several places that the max-norm is a tighter approximation of the rank than the nuclear norm. To the best of my knowledge, this is imprecise: It's a different relaxation, using a different underlying norm assumption on the latent factors (see for instance the introduction in the cited paper [13]). At most, one can argue that it appears to be a better approximation in many cases. - It's hard to see from Figure 1 that OMRMD is better than OR-PCA: They both look more or less the same.


Spatio-temporal Representations of Uncertainty in Spiking Neural Networks
It has been long argued that, because of inherent ambiguity and noise, the brain needs to represent uncertainty in the form of probability distributions. The neural encoding of such distributions remains however highly controversial. Here we present a novel circuit model for representing multidimensional real-valued distributions using a spike based spatio-temporal code. Our model combines the computational advantages of the currently competing models for probabilistic codes and exhibits realistic neural responses along a variety of classic measures. Furthermore, the model highlights the challenges associated with interpreting neural activity in relation to behavioral uncertainty and points to alternative populationlevel approaches for the experimental validation of distributed representations. Core brain computations, such as sensory perception, have been successfully characterized as probabilistic inference, whereby sensory stimuli are interpreted in terms of the objects or features that gave rise to them [1,2]. The tenet of this Bayesian framework is the idea that the brain represents uncertainty about the world in the form of probability distributions. While this notion seems supported by behavioural evidence, the neural underpinnings of probabilistic computation remain highly debated [1,2]. Different proposals offer different trade-offs between flexibility, i.e. the class of distributions they can represent, and speed, i.e. how fast can the uncertainty be read out from the neural activity. Given these two dimensions, we can divide existing models in two main classes.
The first set, which we will refer to as spatial codes, distributes information about the distribution across neurons; the activity of different neurons reflects different values of an underlying random variable (alternatively, it can be viewed as encoding parameters of the underlying distribution [1, 2]). Linear probabilistic population codes (PPCs) are a popular instance of this class, whereby the log-probability of a random variable can be linearly decoded from the responses of neurons tuned to different values of that variable [3]. This encoding scheme has the advantage of speed, as uncertainty can be decoded in a neurally plausible way from the quasi-instantaneous neural activity, and reproduces aspects of the experimental data. However, these benefits come at the price of flexibility: the class of distributions that the network can represent needs to be highly restricted, otherwise the network size scales exponentially with the number of variables [1].
This limitation has lead to a second class of models, which we will refer to as temporal codes.These use stochastic network dynamics to sample from the target distribution [4,1]. Existing models from this class assume that the activity of each neuron encodes a different random variable; the network explores the state space such that the time spent in any particular state is proportional to its probability under the distribution [4]. This representation is exact in the limit of infinite samples. It has several important computational advantages (e.g. easy marginalization, parameter learning, linear scaling of network size with the number of dimensions) and further accounts for trial-totrial variability in neural responses [1]. These benefits come at the cost of sampling time: a fair representation of the underlying distribution requires pooling over several samples, i.e. integrating neural activity over time. Some have argued that this feature makes sampling unfeasibly slow [2].
Here we show that it is possible to construct spatio-temporal codes that combine the best of both worlds. The core idea is that the network activity evolves through recurrent dynamics such that samples from the posterior distribution can be linearly decoded from the (quasi-)instantaneous neural responses. This distributed representation allows several independent samples to be encoded simultaneously, thus enabling a fast representation of uncertainty that improves over time. Computationally, our model inherits all the benefits of a sampling-based representation, while overcoming potential shortcomings of classic temporal codes. We explored the general implications of the new coding scheme for a simple inference problem and found that the network reproduces many properties of biological neurons, such as tuning, variability, co-variability and their modulation by uncertainty. Nonetheless, these single or pairwise measures provided limited information about the underlying distribution represented by the circuit. In the context of our model, these results argue for using decoding as tool for validating distributed probabilistic codes, an approach which we illustrate with a simple example.

The authors present a recurrent spiking neural network model for sampling from multivariate probability distributions through MCMC. A key advantage of their approach is to separate computation and representation: Samples of the target distribution are obtained from the network response via a linear decoder, i.e. neurons do not need to directly correspond to random variables. This gives rise to a rich distributed spatio-temporal code to represent the underlying distribution. Furthermore, the resulting flexibility in the network architecture allows to run multiple MCMC chains in parallel and, thus, to approximate the target distribution near-instantaneously.The theory is complemented by computer simulations that (a) demonstrate the general feasibility of the approach, (b) illustrate how the network performs inference (in a simple generative model) and (c) compare response characteristics with findings from experimental neuroscience. Finally, the authors explore how the approach can be used to identify (properties of) the underlying distribution when the linear decoder is not known; an important step for an application to experimental data.Overall, this is a good manuscript. It contributes several new ideas which are of interest to a broader research community. The manuscript is mostly very well written (particularly the introduction and discussion), the computer simulations and figures illustrate the network properties well. Finally, the work perfectly fits the scope of the conference.Nonetheless, I have some critical remarks which reduced the otherwise positive impression.(A) The manuscript suffers from missing or extra words which, in some cases, even affect the content. E.g. in lines 46, 47, 142, 153, 188, 223 and 386.(B) The spike trains o_i are not defined in line 80. The reset mechanism (described in [6]) is not mentioned around line 140.(C) The derivation is highly compressed and definitely demands reading of refs [6-8]. Also the SI contains relevant information (e.g. the generative model definition). Furthermore, I didn't find important parameters of the simulations (e.g. number of neurons and decoding matrix in Fig 1, lambda, tau_slow,...). This renders a verification of the results almost impossible. Some details to the simulations should be added to the SI.(D) While the network supports a high degree of freedom in the recurrent weight matrices, connections are still symmetric (at least for the Gaussian case). Such limitations should be mentioned at some point.(E) I was wondering if all equations are fully correct. I didn't delve into the details, so the following are just some points the authors might want the check. In line 141, shouldn't the summation run over the first index of the decoder? The sparseness parameter lambda changes the spike response but does not enter any other equation to account for this effect. Is this correct? Similarly, the time constant tau_slow that scales the recurrent weights and drift is not further specified. Can it be chosen arbitrarily, or is this tau_v?In summary, I had the impression that a bit too many different research questions were put into this nine pages manuscript. This comes at the cost of reduced clarity of the individual parts. For instance, since the derivation rests upon some approximations, it would be helpful to explore the range of validity. Also the intriguing multi-chain sampler would have deserved a more comprehensive presentation. Still, I consider this manuscript an important and valuable contribution to the conference if the authors address some of the above issues, and I hope that (one or more) follow-up papers can investigate the presented ideas in greater detail.


Hamming Ball Auxiliary Sampling for Factorial Hidden Markov Models
We introduce a novel sampling algorithm for Markov chain Monte Carlo-based Bayesian inference for factorial hidden Markov models. This algorithm is based on an auxiliary variable construction that restricts the model space allowing iterative exploration in polynomial time. The sampling approach overcomes limitations with common conditional Gibbs samplers that use asymmetric updates and become easily trapped in local modes. Instead, our method uses symmetric moves that allows joint updating of the latent sequences and improves mixing. We illustrate the application of the approach with simulated and a real data example.

This paper introduces a new sampling algorithm for Factorial Hidden Markov models. The standard algorithm for sampling the hidden chains in any HMM type of model is the forward filtering backward sampling algorithm. For many HMM like models this works very well; however for the FHMM, exact sampling from the joint distribution of all random variables in the hidden chain takes exponential time. One possible solution is to sample one or a few chains at a time keeping the other chains fixed. The problem with this approach is that this doesn't easily allow for multiple changes in multiple chains in one sampling pass.This paper introduces a new algorithm called Hamming Ball Auxiliary Sampling. The idea of this sampling algorithm, is starting from the current sample X of all variables of the hidden chain, to introduce an auxiliary variable U which is a sample from the Hamming ball of a certain radius from the current sample where the Hamming ball is defined as any time step of all joint chain variables to be within Hamming distance m from the current sample. Next, the sample X is resampled conditional on U again by condition on being Hamming distance m away from U. As such, between two samples of X, at least 2*m bits can be flipped.The paper is very well written and explains the algorithm well. There is a good empirical evaluation.From experience, one problem with sampling from FHMM's is that if two parallel chains have very similar states, then it is likely that the parameters for those two chains will be very similar. Essentially, what should be one feature ends up being duplicated. This sampling algorithm does not make it more likely that this local maxima is escaped. I think it is still important to point out that that is still an issue (say around line 126).l197: My only real complaint about the paper is that I'd like to see some algorithmic details on how this sampler can be efficiently implemented. I'm curious to know if the authors use an efficientl algorithm to sample from the Hamming ball? I.o.w. I'd add a section on implementation (and really I'd like to see some code published).l215 - l230: I think this can easily be made an appendix or extension of the paper. It's a bit trivial and doesn't really make the paper that more impressive.Section 4.1 As you pointed out, the different algorithms have different time complexities. I'd like to see Figure 3 scaled w.r.t. real complexity (rather than iterations which gives the HB algorithms an unfair advantage).Nice work!


A Safe Screening Rule for Sparse Logistic Regression
The 1 -regularized logistic regression (or sparse logistic regression) is a widely used method for simultaneous classification and feature selection. Although many recent efforts have been devoted to its efficient implementation, its application to high dimensional data still poses significant challenges. In this paper, we present a fast and effective sparse logistic regression screening rule (Slores) to identify the "0" components in the solution vector, which may lead to a substantial reduction in the number of features to be entered to the optimization. An appealing feature of Slores is that the data set needs to be scanned only once to run the screening and its computational cost is negligible compared to that of solving the sparse logistic regression problem. Moreover, Slores is independent of solvers for sparse logistic regression, thus Slores can be integrated with any existing solver to improve the efficiency. We have evaluated Slores using high-dimensional data sets from different applications. Experiments demonstrate that Slores outperforms the existing state-of-the-art screening rules and the efficiency of solving sparse logistic regression can be improved by one magnitude.

In this paper, the authors propose a new sparse logistic regression screening rule (Slores) for feature selection, which discard the features with zero coefficients by solving a convex optimization problem with closed form solution.  A series of evaluations have been conducted to show its effectiveness.This idea is simple but interesting and I like it. However, I still have some questions:1.	In the section 3, the authors propose to estimate the upper bound by solving a optimization problem, where the critical parameter $\lambda_0$ and $\theta_{\lambda_0}^\ast$ are set to be $\lambda_\max$ and $\theta_{\lambda_{max}}^\ast$ respectively. However, how can we know $\lambda_\max$ and $\theta_{\lambda_{max}}^\ast$ at the beginning?2.	In the 138th row, a “|” is missing in the equation.3.	In the experimental results, the authors just use the reject ratio to measure the performance. However, I still have a problem: whether the non-zero coefficients can be affected by discarding these features? Some other feature selection measures are recommended. 4.	The authors should compare the proposed method with the state-of-the-art, such as “ Lasso Screening Rules via Dual Polytope Projection, ICML, 2013”.


Recursive Inversion Models for Permutations
We develop a new exponential family probabilistic model for permutations that can capture hierarchical structure and that has the Mallows and generalized Mallows models as subclasses. We describe how to do parameter estimation and propose an approach to structure search for this class of models. We provide experimental evidence that this added flexibility both improves predictive performance and enables a deeper understanding of collections of permutations.

Mallows models are a classically studied class of distributions overpermutations that can be viewed as a sequential model in which itemsare inserted one by one into a ranking.  This paper proposes an interestinghierarchical generalization of Mallows models in which groups of itemsare sequentially ``merged'' together (as they would be in mergesort).The model can also be viewed as a special case of a recently proposedclass of ``riffle independent'' models by Huang/Guestrin, but witha more tractable number of parameters in general and better computationalproperties.There are several nice contributions in this paper, including a simpleand elegant characterization of identifiability of the structure, as well asan interesting structure estimation algorithmbased on the inside-outside parsing algorithm for stochastic context freegrammars.The authors also discuss a few results on Sushi and Irish election datasetswhich seem promising.  I think that the tree visualizations of the distributionsare particularly intuitive, and should be an easy way to do aa preliminary analysis of any new ranking dataset.I would have been interested in the synthetic data experiments too, but these did not seem discussed.Though now there may be a lot of new space for the authors to discuss these (see below).It is interesting that the authors were able to reproduce the clusters that were found inthe Huang/Guestrin analysis --- which suggests a much simpler form of interaction wassufficient to capture most of the variation.  I would love to see a discussion ofwhat cannot be captured well by the proposed RIM model.Overall, the paper is well written, highly technical, and treatsan interesting class of models that should have broad applicationin ranking.   Given the structure of the model, this paper should alsolead to a number of easy extensions including mixtures models andlearning with partially ranked data (in the style of the Lu/Boutilier,Huang et al, or Lebanon/Mao papers).I heartily recommend acceptance to NIPS.some small errors:grep for: paramter, matricies, ``If if'', appraoch, interpretabtionThe algorithm for SASEARCH seems to appear twice in the paper.The authors could get significantly more space by deleting one :)


Deterministic Symmetric Positive Semidefinite Matrix Completion
We consider the problem of recovering a symmetric, positive semidefinite (SPSD) matrix from a subset of its entries, possibly corrupted by noise. In contrast to previous matrix recovery work, we drop the assumption of a random sampling of entries in favor of a deterministic sampling of principal submatrices of the matrix. We develop a set of sufficient conditions for the recovery of a SPSD matrix from a set of its principal submatrices, present necessity results based on this set of conditions and develop an algorithm that can exactly recover a matrix when these conditions are met. The proposed algorithm is naturally generalized to the problem of noisy matrix recovery, and we provide a worst-case bound on reconstruction error for this scenario. Finally, we demonstrate the algorithm's utility on noiseless and noisy simulated datasets.

PAPER SUMMARYIn this paper, the authors propose a model and an algorithm for completing a symmetric, positive semidefinite matrix. Unlike prior work that assumes that the observed entries are drawn uniformly at random, this work assumes that the observed entries come from deterministic collection of its principal sub matrices. The paper derives either necessary or sufficient conditions on these sub matrices that guarantee the perfect recovery of the unknown entries. The paper also considers the case where the data is corrupted by noise and derives an upper bound on the error between the estimated and the true matrix.  Simulations on synthetic data verify the correctness of the proposed model.SPECIFIC COMMENTSThis paper is well-organized and mathematically rigorous. The theoretical analysis is novel since it addresses deterministic matrix completion model with specific structure. In particular, rather than addressing the well studied case where the observed entries are drawn uniformly at random, the paper assumes that the observed entries are drawn from a set of principal sub matrices that satisfy certain properties. Perfect matrix completion can be achieved in the noiseless case, while an upper bound is derived for the noisy case. Overall, I find the results in this paper to be very novel. I only have a few minor comments below that can help improve clarity. My most important comment is that the experiments can be improved by adding a comparison with other methods as well as experiments on real data.1) The term "minor" usually refers to the "determinant of some smaller square matrix". In this paper, the term is used to refer to the sub matrix itself without the determinant. I suggest the authors use a different name to avoid confusion.2) Some of the notation is confusing. For example, C_l and C(\rho_l, :) are both used to refer to the same matrix, which is confusing to the reader.3) The proposed model and algorithm are based on several assumptions on the principal sub matrices of the data matrix. In practice, it is not clear if these assumptions are satisfied in real world applications. For example, given a dataset, how does one derive the collection of principal sub matrices  that satisfy assumptions A1-A3? Or if these assumptions fail, is there a way of permuting the rows and columns of the given matrix so that the assumptions are satisfied?4) In the experiments, the authors did simulations on synthetic data to verify the correctness of the algorithm. However there is no comparison with other matrix completion methods, which is not convincing enough to show the advantage of the proposed algorithm. It would have been nice to see an example where standard matrix completion algorithms fail, but the propose algorithm succeeds. Also, the paper would be strengthened if there were simulations on real data. One specific example that comes to mind is motion segmentation in computer vision, where the missing entries occur due to appearance and disappearance of moving objects in a video. In that case, principal sub matrix patterns show up naturally.


Fast and Robust Least Squares Estimation in Corrupted Linear Models
Subsampling methods have been recently proposed to speed up least squares estimation in large scale settings. However, these algorithms are typically not robust to outliers or corruptions in the observed covariates. The concept of influence that was developed for regression diagnostics can be used to detect such corrupted observations as shown in this paper. This property of influence -for which we also develop a randomized approximation -motivates our proposed subsampling algorithm for large scale corrupted linear regression which limits the influence of data points since highly influential points contribute most to the residual error. Under a general model of corrupted observations, we show theoretically and empirically on a variety of simulated and real datasets that our algorithm improves over the current state-of-the-art approximation schemes for ordinary least squares.

This paper looks at alternatives to LS for solving linear regressionproblems.  Classical statistics will consider a score function as away of weighting observations.  But, these would only capture anadditive error which is different than Gaussian.  This paper instead,also considers the influence of a point.  Statisticians for years havethrown out influential outliers--but as	far as I can find, this	is thefirst automatic	procedure that	uses influence as a way	ofdown-weighting a	regression.The probabilistic model	presented here is not the driving force	of thepaper. 	In other words,	it is pretty easy to identify which points arecontaminated and which are clean.  (This is very easy for large psince the two distributions won't overlap.)  So	if the estimatorpresented was tied to this model, I would be unexcited about theresults.  But, the estimator is	fairly natural.	 So the	model is onlybeing used as an example.1) Put a line where the accuracy of the MLE for your probabilisticmodel would be.  You shouldn't end up being close to this line sinceyou aren't assuming your model.  But it still is a more usefulcomparison point than the LS line.  If you don't want to bother	withdoing the de-convolution problem, you can assume that the ML hasaccess to the random variables U.  Then	it will	be a lower bound onhow well it will perform.  As p	--> infinity, this lower bound willbecome tight.2) Make the connection to some of the literature on the use of scoringfunctions (Since this goes back to the early 1970's I'm not going tosuggest any literature).  For example, the aRWS can be thought of asdown weighting points by a 1/epsilon^2.  If one were to down weight by1/epsilon, that would correspond to doing a L-1 regression.  So itwould be similar to assuming a double exponential distribution for theerrors.  Your weighting function is even more draconian.  So whatdistribution does it correspond to if you think of it as a score?(I'm guessing it looks like a Cauchy distribution--but onlyapproximately.)  (Note: you are sampling, whereas I'm more used tothinking of weights.  So it might be that your 1/epsilon^2 sampling isrelated to a 1/epsilon weight.  If so, then it is approximating a L-1regression.  This would be very nice to point out if it were true.)3) Are you claiming to be the first people to suggest weighting by1/influence?  This might be true--but it is a strong claim.4) I think what you are doing is using a different regression functionfor "large observations" than you use for "small observations."  Oneway to test this and provide a better comparison for LS would be todefine tilde-l as you do and then interact this variable with all yourX's in your regression.  This will allow the LS methods to have accessto tilde-l.  Hence LS should be able to end up with a better estimatorsince it now can have different slopes for the pure X observationsthan it has for the X+W observations.5) There is extensive literature on both errors in variables androbust regressions.  At least a few of these 1000+ papers should havesomething useful to say.  Put in at least some effort into findingsomething that connects.  That will help the implied claim that yourmethods are in fact new.6) PLease look at this 1980's NBER paper: http://www.nber.org/chapters/c11698.pdf In particular, equations (15) and (16).  This is very close to the estimator you are using.  It would be nice if you were to chase down some of the modern references and see if there are any connections to your work.


Sparse Bayesian structure learning with dependent relevance determination prior
In many problem settings, parameter vectors are not merely sparse, but dependent in such a way that non-zero coefficients tend to cluster together. We refer to this form of dependency as "region sparsity". Classical sparse regression methods, such as the lasso and automatic relevance determination (ARD), model parameters as independent a priori, and therefore do not exploit such dependencies. Here we introduce a hierarchical model for smooth, region-sparse weight vectors and tensors in a linear regression setting. Our approach represents a hierarchical extension of the relevance determination framework, where we add a transformed Gaussian process to model the dependencies between the prior variances of regression weights. We combine this with a structured model of the prior variances of Fourier coefficients, which eliminates unnecessary high frequencies. The resulting prior encourages weights to be region-sparse in two different bases simultaneously. We develop efficient approximate inference methods and show substantial improvements over comparable methods (e.g., group lasso and smooth RVM) for both simulated and real datasets from brain imaging.

I appreciate the thorough author feedback and find the authors' arguments mostly compelling.  I have decided to increase my review score to be above the acceptance threshold under the understanding that the authors will clarify the technical concerns in a final paper and expand on the bibliography.  I am still not convinced that the covariance function is positive semi-definite for inputs with dimension greater than one.  I understand that as the inputs correspond to the position of the weights they will generally will be of low dimension.  However, it is much more interesting to allow the weights to be located in 2D (images), 3D (videos) or even a higher dimensional latent space.  As a simple remedy, the authors could make the covariance be the product over dimensions as in [11] or they could restrict the dimensionality of the inputs to be one.  Otherwise, I would like more justification in the manuscript or in an appendix that the proposed covariance is PSD.------------------This paper explores the problem of linear regression in the scenario where many of the weights are expected to be zero.  The innovation of the paper is the development of a hierarchical Bayesian model for which this sparsity follows a spatial pattern, the sparsity of the weights is dependent on the relative spatial location of each weight.  The sparsity pattern is achieved through placing a Gaussian process prior on the regression weights.  Specifically, the authors place a GP prior, conditioned on the spatial location of the weights, over a latent variable governing the diagonal of a covariance matrix.  This is composed with a smoothness covariance in such a way that the latent variables govern the magnitude of the eigenvalues.  After fitting the hyperparameters of the model, the authors prune the weights corresponding to small eigenvalues of the covariance, achieving the desired sparsity.The paper is well written, structured and easy to understand.  The authors do a good job of motivating the problem and sparse regression is certainly a topic of interest to the NIPS community.  The model seems novel and interesting and the experiments, although very limited, do demonstrate the quality of the approach.  There are, however, some concerns with the methodology and the correctness of the model and the empirical evaluation is not exhaustive enough to be compelling.Although the authors start with a neat model, the approach seems to turn into a rather heuristic procedure due to scalability, tractability and numerical issues.  Although I empathize with the authors, the Bayesian model becomes an approximation to an approximation of the original model.  The paper seems like a very good start to an interesting model, but the approach seems a little suspect (i.e. PSD-ness, pruning weights corresponding to small eigenvalues, inverting with the SVD due to a singular covariance, various approximations) and thus I would like significantly more justification of this method both in the text and empirically.A major concern is the chosen covariance function (equation 5).  Specifically, I doubt that it is positive definite and I find it odd in general.  Why would one have a periodic covariance over the location of the weights?  This seems strange as some closer weights will be anticorrelated while farther weights will be correlated.  I think this is only positive definite if the distance between the location of the weights i and j is bounded between [pi/2, -pi/2].  Perhaps this is a reason for the numerical instability that is referred to on line 208?  The authors must justify that this covariance function is a reasonable choice (i.e. show that it is PSD) and provide some reasoning.The authors should expand significantly on the related work and references.  There is a wealth of literature about structured sparse regression from recent work in submodularity and convex optimization (see e.g. Bach, http://arxiv.org/abs/1111.6453), and in the machine learning literature in general (e.g. Huang et. al http://jmlr.org/papers/volume12/huang11b/huang11b.pdf).  There is plenty of space on the references page to add these.Radford Neal and David Mackay should really be referenced for automatic relevance determination (ARD).In the abstract, the authors suggest that the Gaussian process prior is conditioned on the Fourier coefficients of the weights.  I'm not sure I understand how the Fourier coefficients come into play here and would appreciate some clarification.There is no analysis done on the impact of the various approximations on the posterior of the model.  Specifically, how is the predictive variance of the model affected with respect to the approximations?  Do they negatively affect the uncertainty of the model (I think they probably do significantly).  I find it strange that the smooth RVM is left out of the final experiment.  This should be included to make the empirical analysis compelling.  However, at least some of the wealth of existing structured sparsity work should also be compared to.  Even if that work is not Bayesian, it give a better perspective on how difficult the problem is.  Finally, it seems to me that Gaussian processes should really be included in the comparison.  Why are these left out?  It doesn't seem that the proposed model scales beyond what a GP can model.


Information-based learning by agents in unbounded state spaces
The idea that animals might use information-driven planning to explore an unknown environment and build an internal model of it has been proposed for quite some time. Recent work has demonstrated that agents using this principle can efficiently learn models of probabilistic environments with discrete, bounded state spaces. However, animals and robots are commonly confronted with unbounded environments. To address this more challenging situation, we study informationbased learning strategies of agents in unbounded state spaces using non-parametric Bayesian models. Specifically, we demonstrate that the Chinese Restaurant Process (CRP) model is able to solve this problem and that an Empirical Bayes version is able to efficiently explore bounded and unbounded worlds by relying on little prior information.

In their paper “Information-based learning by agents in unbounded state spaces” the authors extend a previous model of information-based exploration as described in reference [11] to unbounded state spaces by introducing a Chinese restaurant process to model transition probabilities.Previous studies have used the Chinese restaurant process for reinforcement learning—for example, reference [2] cited by the authors. It would therefore be good if the authors could clarify the differences to previous studies that have used Chinese restaurant processes in reinforcement learning to clarify originality.L 130: verb is missing in the second part of the sentenceTo compute the information gain the authors need to compute relative entropies between the true state transition distribution and the estimated state transition distribution. As explained by the authors, these two distributions initially have different dimensionality. To make the distributions commensurable they expand the dimensionality of the estimated distribution to include states that have not been observed to date by uniform distributions. However, this seems to defeat the purpose of the paper to some extent, since the true cardinality of all random variables has to be known in equation (9) and all subsequent equations (10)-(13). This is a crucial question to clarify.L 160Question wrt equation (12)The last \hat{\theta}_sa is the updated model? In that case shouldn’t it be written as \hat{\theta}^{a,s\rightarrow s^*} as in equation (6) in reference [11]?The information gain measure the authors use in equation (13) is somewhat unconventional. Therefore, it would be important to see how it performs in the most simple exploration problem—that is the bandit problem. This should be even simpler to simulate than the experiments they present. For bandit problems there are well established bounds on how efficient different exploration methods are and the authors could compare against established schemes like UCB or Thompson sampling.


Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation
This paper proposes a new hybrid architecture that consists of a deep Convolutional Network and a Markov Random Field. We show how this architecture is successfully applied to the challenging problem of articulated human pose estimation in monocular images. The architecture can exploit structural domain constraints such as geometric relationships between body joint locations. We show that joint training of these two model paradigms improves performance and allows us to significantly outperform existing state-of-the-art techniques.

TECHNICAL CONTENT & CLARITY:The "deep convolutional network" part-detectors of Sec. 3.1 seem like a sensible application of recently developed deep learning methods.  There does not appear to be significant novelty, but the architecture seems carefully engineered and is clearly described.  One comment, if the "approximate" architecture in Figure 4 is used for the actual results, why bother to show the more complex architecture in Figure 3?  There are no conceptual principles presented as to why one should be better than the other; the validation is experimental, so why not focus your explanatory effort on the thing you actually experiment with?In Sec. 3.2, there is no valid joint distribution defined on the body part locations, and no non-trivial graphical structure; thus it seems inappropriate to call this a "graphical model".  Instead there is a single round of message passing between all part pairs, as illustrated in Figure 5.  This message passing has a form motivated by the sum-product BP algorithm, but need not correspond to BP (for example in a valid graphical model there would be constraints relating f|s and s|f which are not enforced here), and is not iterated to any mutually-consistent fixed point.  A comment after Equation (2) says this formulation is not "exactly equivalent to an MRF", but I would argue more strongly, and say that the final formulation is rather distant from an MRF.  The claims in earlier parts of the paper on integration with graphical models should be moderated to reflect this.For related work which is more similar to this approach, see the "cascaded classification models" of Heitz 2008, or more distantly the "sum-product networks" developed by Domingos and colleagues.  The approach is rather different from work like Bergtholdt 2010, which carefully treats the issues of inference and learning in fully-connected part-based graphs.EXPERIMENTS:Results on two contemporary pose estimation datasets are very good, improving on several strong baselines.  A careful internal comparison also shows relative benefits of adding the final spatial message-passing layer, and doing a final round of back-propagation training to refine the full set of model parameters.  These results are certainly the greatest strength of the paper.I would have liked to also see an evaluation on the "Buffy the Vampire Slayer" dataset, which has been very widely studied in the pose estimation literature, and would this provide a wider set of baselines.REFERENCES:G. Heitz, S. Gould, A. Saxena, and D. Koller (2008). "Cascaded Classification Models: Combining Models for Holistic Scene Understanding." Advances in Neural Information Processing Systems (NIPS 2008).Bergtholdt, Kappes, Schmidt, Schnorr, A Study of Parts-Based Object Class Detection Using Complete Graphs.  International Journal of Computer Vision, Volume 87 Issue 1-2, March 2010, Pages 93 - 117.


Parallel Direction Method of Multipliers
We consider the problem of minimizing block-separable (non-smooth) convex functions subject to linear constraints. While the Alternating Direction Method of Multipliers (ADMM) for two-block linear constraints has been intensively studied both theoretically and empirically, in spite of some preliminary work, effective generalizations of ADMM to multiple blocks is still unclear. In this paper, we propose a parallel randomized block coordinate method named Parallel Direction Method of Multipliers (PDMM) to solve optimization problems with multi-block linear constraints. At each iteration, PDMM randomly updates some blocks in parallel, behaving like parallel randomized block coordinate descent. We establish the global convergence and the iteration complexity for PDMM with constant step size. We also show that PDMM can do randomized block coordinate descent on overlapping blocks. Experimental results show that PDMM performs better than state-of-the-arts methods in two applications, robust principal component analysis and overlapping group lasso.

This paper considers the problem of minimizing block-separable functions subject to linear constraints. A variant of ADMM (called PDMM) is proposed to deal with the multiple blocks case in this paper: it randomly selects K blocks of the primal variable to update and uses a backward step to refine the dual variable; the process to update the primal variable can be implemented in parallel. The sublinear convergence rate is proved for the proposed method. This method generalized several variants of ADMM.This paper is well written and easy to follow. I am positive to accept this paper. My questions and comments are given below:1) The algorithm description in (5) and (6) is not clear enough. Do you update all coordinates of y^{t+1} and \hat{y}^{t+1} or just a single block "i"? My understanding is that you only update a block of the dual variable "y", but I did not find how to select "i". This should be clarified in the revision.2) In Theorem 2, I believe that "\sum_{i=1}^I" was missing behind of "{" from your proof.3) More discussion about Theorem should be included. Theorem 2 (the main result in this paper) provides the convergence rate of the proposed method PDMM. First, PDMM generalizes several variants of ADMM. A natural question is if Theorem 2 (by properly choose parameters like K, J) is consistent with the convergence rates of those variants. Second, since you split the transformation matrix A into multiple blocks, what is the optimal way to split it from your theorem?4) I am curious of the comparison on the tensor completion problem ("tensor completion for estimating missing values in visual data, 2012"), which also has the multiple block structure. Do you have any clue which variant of ADMM is optimal?    Minors:1) In (9), a space is missing behind of "min"2) Line 217, remove the space before "Section"


Spectral Methods for Supervised Topic Models
Supervised topic models simultaneously model the latent topic structure of large collections of documents and a response variable associated with each document. Existing inference methods are based on either variational approximation or Monte Carlo sampling. This paper presents a novel spectral decomposition algorithm to recover the parameters of supervised latent Dirichlet allocation (sLDA) models. The Spectral-sLDA algorithm is provably correct and computationally efficient. We prove a sample complexity bound and subsequently derive a sufficient condition for the identifiability of sLDA. Thorough experiments on a diverse range of synthetic and real-world datasets verify the theory and demonstrate the practical effectiveness of the algorithm.

This paper addresses the classical problem of unsupervised learning of latent topic model, with an extra variable called response, which can be a score. The main issue is that the topic model (and the embeddings deduced from it) may not help in learning this extra variable, as the response can be induced by phenomena that are orthogonal to the topics. The goal of the so-called supervised topic model learning is to drive the topic learning into a direction which makes it useful w.r.t. the prediction of this extra variable (by a regression).The basic model considered is the Latent Dirichlet allocation (LDA) model. The parameters of such a model can be estimated with Max-Likelihood, sampling, or recently spectral methods (methods of moments). The spectral method is based on the consistent estimate of a finite number of observable statistical moments, from which it is possible to recover the parameters thanks to simple linear algebra operations (in the original paper) or more complex ones (like tensor decomposition). Starting from previous work done on spectral learning of LDA, the paper shows how the same method  can be used to learn the response variable, by adding an extra family of observable moments, and by showing how it is possible to recover the parameters of the regression via tensor decomposition.Some experiments show that the algorithm is good w.r.t. other methods in parameter recover  on a synthetic dataset. Conversely, the method does not perform well in real data experiments, with a max. likelihood objective. This is not surprising, as the non-spectral method usually focus on direct maximization of likelihood, whereas spectral does not. Also, the spectral method does not behave well in general with small samples (compared to the information needed for a good estimate)The paper is well written, the mathematical part is solid. The algorithm described in the paper, to my opinion, is more an extension of the spectral algorithm for LDA than a real new method. Its main interest is to show that the spectral method can be also applied to the sLDA model. More work needs to be done to make this method efficient in real data problems (as it is for spectral method in general).(I think that when the authors say that « Theorem 1 is that it provides a necessary condition for a supervised LDA model to be identifiable », they mean in fact « sufficient », right?)Pros:- good adaptation of the spectral LDA learning algorithm- good theoretical work- theoretical boundsCons: - more an extension of known methods than a new contribution- not convincing in real data learning