Projecting Markov Random Field Parameters for Fast Mixing
Markov chain Monte Carlo (MCMC) algorithms are simple and extremely powerful techniques to sample from almost arbitrary distributions. The flaw in practice is that it can take a large and/or unknown amount of time to converge to the stationary distribution. This paper gives sufficient conditions to guarantee that univariate Gibbs sampling on Markov Random Fields (MRFs) will be fast mixing, in a precise sense. Further, an algorithm is given to project onto this set of fast-mixing parameters in the Euclidean norm. Following recent work, we give an example use of this to project in various divergence measures, comparing univariate marginals obtained by sampling after projection to common variational methods and Gibbs sampling on the original parameters.
The principal contributions of this paper are, first, a set of sufficient conditions to guarantee that univariate Gibbs sampling on an MRF will be fast-mixing (Section 4), and an algorithm to project onto this set in the Euclidean norm (Section 5). A secondary contribution of this paper is considering an alternative matrix norm (the induced ∞-norm) that is somewhat looser than the spectral norm, but more computationally efficient. Following previous work [3], these ideas are experimentally validated via a projected gradient descent algorithm to minimize other divergences, and looking at the accuracy of the resulting marginals. The ability to project onto a fast-mixing parameter set may also be of independent interest. For example, it might be used during maximum likelihood learning to ensure that the gradients estimated through sampling are more accurate.

The paper considers the problem of marginal inference in discreteMRFs. It characterizes conditions on the MRF parameters under whicha Gibbs sampler will be rapidly mixing. Further, it outlines aprocedure by which the parameters of any discrete MRF can beprojected into this family of MRFs. Some preliminary experimentalresults are provided to illustrate the method.The paper makes an important contribution to approximate marginalinference in discrete MRFs. The ideas presented are clear and infact quite simple and natural in hindsight - which is a strength ofthe work. The approach is unique, but draws on existing ideas, such as approximate divergence computation usinglow tree-width sub-graphs and Gibbs sampling.There are several aspects of the work which need clarification -the suggestions below are to help improve the presentation, make itaccessible to a wider audience, and connecting it with otherrelevant related work:i) The idea of considering a smoothed approximation to a non-smoothproblem has been studied in the literature and there are existingresults, e.g., seeYurii Nesterov. Smooth minimization of non-smooth functions. Math.Program., 103(1):127–152, 2005.The characterization and discussion in Section 5 can be improvedusing this literature. In particular, for suitable variants of (4),one may be able to characterize (e.g., give bounds on) the natureof the approximation.ii) A brief comment on how the problem in Section 5.1 satisfies theconditions of Danskin's theorem will be helpful to readers. Also,please add a citation.iii) Section 6.2.2 is interesting. Some discussion on how to choose{\cal T} will be helpful (there is a discussion for the specificexperiments). For example, I wonder if one may be able to choose{\cal T} to be all spanning trees and get away with tractablecomputations using spanning tree LPs.iv) The work is close in spirit to [2], so a brief discussion onwhat aspects are different and/or does not directly follow from theIsing case will be helpful.v) Related to the above point - all experiments seem to be on Isingmodels, which is covered in [2]. Some evidence of performance ongeneral discrete MRFs is needed.vi) The experimental results are promising, but was hard to followin the first read. The proposed methodology works in two stages -approximate the parameters, then run Gibbs sampling. A discussionon the break-up of the run-time between these stages will behelpful. A study of the trade-off between the number of gradientsteps in the first stage and the number of Gibbs sampling steps inthe second stage will be instructive.vii) Over the last few years, a new approach to marginal inferencehas been studied - based on solving suitable random MAP inferenceproblems, e.g., see the following and their referencesT. Hazan and T. Jaakkola. On the partition function and randommaximum a-posteriori perturbations. ICML, 2012.T. Hazan, S. Maji, and T. Jaakkola. On sampling from the Gibbsdistribution with random maximum a posteriori perturbations. NIPS,2013.A discussion and comparison with the random-MAP inference approachwill make the paper stronger.viii) The writing is in bad shape, and seems to have been done in ahurry. Please do a careful pass to fix the flow and presentation.


A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights
We derive a second-order ordinary differential equation (ODE), which is the limit of Nesterov's accelerated gradient method. This ODE exhibits approximate equivalence to Nesterov's scheme and thus can serve as a tool for analysis. We show that the continuous time ODE allows for a better understanding of Nesterov's scheme. As a byproduct, we obtain a family of schemes with similar convergence rates. The ODE interpretation also suggests restarting Nesterov's scheme leading to an algorithm, which can be rigorously proven to converge at a linear rate whenever the objective is strongly convex.

==========added after author rebuttal============Re Theorem 4.3. I think my initial comment is correct but maybe I should have been more explicit: Eq. (36) in Tseng's paper or the similar stepsize in Beck and Teboulle need not be satisfied with equality. When one bounds the suboptimality wrt a global minimizer, such as the f* in the author's Theorem 4.3, the inequality <= in Tseng's Eq. (36) is enough, which makes the stepsize (r-1)/(k+r-1) feasible. The strict equality is needed when one compares wrt an arbitrary point, such as in the theorem Tseng derived. This is apparent if one checks when Eq. (36) is used in the proof, or when Beck and Teboulle applied their stepsize rule (proof of their Lemma 4.1, 2nd displayed equation in page 195, note that v_k >=0). Re Experiments: The authors seem to misunderstand my comment. I was not asking for a thorough comparison with a great number of competitive algorithms, but a few that naturally pop out in one's mind. These comparisons not only give us a better picture but also provide guidance for others who might be interested in applying the proposed algorithms. ======================================Nesterov's accelerated gradient algorithm has been a very popular algorithm for handling large datasets, and is known to be optimal among first order methods. The authors re-derived Nestorov's algorithm as a discretization of a second order differentiable equation. It appeared the convergence of the continuous ODE is much cleaner to deal with. Moreover, inspired by the continuous ODE, the authors proposed two variations, one of which could exploit strong convexity even when the parameter is not known beforehand. Limited experimental results partially validate the theoretical discoveries. As far as the reviewer is concerned, the convergence results derived in this submission are novel and potentially significant. Although the idea to relate continuous ODE with discrete gradient methods has a long history, the authors have done a superb job in instantiating the general principle to Nesterov's algorithm, overcoming some technical difficulties along the way. Theorem 4.3 is not quite as new as it appears to be. In fact, the update rule in Eq. (12) makes it clear that it is nothing but the same update rule in Eq (34) in the following paper:"Approximation accuracy, gradient methods, and error bound for structured convex optimization, Tseng'10", under the identification theta_k = (r-1) / (k+r-1). In fact, Tseng's treatment is even (slightly) more general, with a very enjoyable proof too. This should be pointed out. The equation in line 260 seems to be new and interesting.Theorem 4.4 is very interesting as it does not require the knowledge of the strong convexity parameter. Unfortunately the current proof appears to be long and tedious. It would be great to polish the proof to further distill the essence. On the other hand, in many machine learning applications the strong convexity indeed is known beforehand (in particular if one uses strongly convex regularizers). The current experiments are not very satisfying. Two very natural comparisons are missing (when strong convexity is present): 1). the usual proximal gradient (without accelerating), which achieves a linear rate with a worse constant; 2). Nesterov's algorithm with the knowledge of the strong convexity parameter. These comparisons will give us a better idea on how effective the restarting strategy is. The experiment suffers also from another apparent issue: Nesterov's original algorithm's oscillating behavior. It is trivial to fix this issue by enforcing monotonicity, i.e., picking y_t as the best (in terms of objective value) of y_t and y_{t-1}. The convergence property will not he jeopardized (because the momentum term is still changing). This is a third competitor the authors should have compared to. Although the authors did give some justification of the restarting strategy, the results are not as convincing as it could be at the moment: 1). Theorem 5.2 is about the continuous ODE and does not allow any nonsmooth component; 2). How does it compare to a simple halving trick, i.e., one starts with a guess mu of the strong convexity parameter and keeps halving it?


On the Number of Linear Regions of Deep Neural Networks
We study the complexity of functions computable by deep feedforward neural networks with piecewise linear activations in terms of the symmetries and the number of linear regions that they have. Deep networks are able to sequentially map portions of each layer's input-space to the same output. In this way, deep models compute functions that react equally to complicated patterns of different inputs. The compositional structure of these functions enables them to re-use pieces of computation exponentially often in terms of the network's depth. This paper investigates the complexity of such compositional maps and contributes new theoretical results regarding the advantage of depth for neural networks with piecewise linear activation functions. In particular, our analysis is not specific to a single family of models, and as an example, we employ it for rectifier and maxout networks. We improve complexity bounds from pre-existing work and investigate the behavior of units in higher layers.

I. Introduction1) Limited number of papers have investigated deep networks from a theoretical perspective?? I disagree with this statement, many papers from Bengio's group do just this. 2) Define what you mean by popular networks, this is too vague (is it CNNs, RNNs, LSTMs, different non-linearities)3) Pascanu did the analysis for ReLU, pls justify the added benefit for doing this for piecewise linear functions? How novel is your work compared to Pascanu-13, as he did this for rectified units. You should be more clear about your novelties compared to this paper.4) Pls quantify what significant improvement means??II. Feedforward Neural Networksa) It woudl be helpful to have some pictures to go along with 2.2, as Pascanu-13 gives, or include this in supplemental material and point to itb) does the idea of input space foldings also depend on the non-linearity or no? this should be stated more clearly?c) So what is the bound on \epsilon to guarantee it stable to perturbation, pls clarify?d) There is very little connection between sections. It would be better if the flow between sub-sections was better.III. Deep Rectifier Networksa) it would be helpful if you had a visual description for 3.1 in the paper (or supplemental material)b) provide a deeper analysis of what the theorem doesIV. Deep Maxout Networksa) again, pictures for this section would be helpful as wellb) a better comparison and analysis for linear regions for rectified and maxout would help the readers understand why one could be preferred over another. This is a weakness of the paperV. Conclusionsa) How do the proofs in Theorem 4 and 8 help to show that deep models generalize better to unseen examples


Do Deep Nets Really Need to be Deep?
Currently, deep neural networks are the state of the art on problems such as speech recognition and computer vision. In this paper we empirically demonstrate that shallow feed-forward nets can learn the complex functions previously learned by deep nets and achieve accuracies previously only achievable with deep models. Moreover, in some cases the shallow nets can learn these deep functions using the same number of parameters as the original deep models. On the TIMIT phoneme recognition and CIFAR-10 image recognition tasks, shallow nets can be trained that perform similarly to complex, well-engineered, deeper convolutional models.

This paper presents experiments with shallow network trained to mimicthe predictions of deeper, more complex networks trained on the samedata.  When trained to predict the pre-softmax outputs of the"teacher" networks, these mimic networks are able to approximate theperformance of the deeper networks, albeit with more parameters.  Theyconsistently outperform networks with the same architecture trainedthe cross-entropy objective function.The paper presents an interesting -- and perhaps useful -- set ofexperiments, but the theoretical analysis is extremely weak.  Thereare a number of bold claims that in my opinion are not demonstrated bythe experimental work, and many speculative statements, the truth ofwhich there has been little effort to investigate.  The paper isreasonably original, though methods similar to the mimic-network havebeen presented before.The paper is clearly written and reasonably well-organised, althoughthe colloquial style ("what is the source of this magic?") in theintroduction is not to my taste.  The strong part of the paper is thedemonstration that a shallower neural network can achieve goodperformance when trained to mimic a deeper "teacher" network, orensemble.  This is good news for software engineers tasked withbuilding low-footprint systems, although, as noted, related techniqueshave already been employed [20].  It is presented as a surprisingfinding, but I am not sure why this should be so: not dissimilarmethods have also had success, such as the KL-HMM.  As in that model,it may be seen that the teacher network is providing a large amount ofinformation to the learner in the vector z(t), as in the KL-HMM wherethe p(t) are used directly, so it is somewhat misleading to say thatthe learner is simply trained to mimic the (one-best) predictions ofthe teacher network - it's being given much more information thanthat.  Analagously, the purported benefit of the KL-HMM is that theposteriors generated by one NN are helpful targets for another (abeitmuch simpler) model.  (Note, I am aware that the KL-HMM differsmaterially from the mimic-model).Section 5.1 is particularly weak in my opinion.  If I were to write itwith my own theories as to why the mimic models work, they would bequite different to those being presented, but no more speculative.The first theory, that the teacher eliminates errors, could easily betested, but isn't.  The second, "the complexity in the data set hasbeen washed away" makes no sense at all -- what does this mean?  Thethird seems more plausible, but again, this could easily be tested."Model compression appears to be a form of regularization" needs muchmore support, as it underpins the core claim that if only shallowmodels could be regularized more successfully, then they could betrained to perform as well as the deep models without the help of ateacher. However, this is a diversion from my main point, which is that Ibelieve that the theoretical premise of the paper is flawed.  Aswidely known and cited [4], a neural network with a single hiddenlayer can approximate any decision boundary, with a large enoughnumber of hidden units.  This point, which seems to be the coreexperimental finding of the paper, is not novel.  However, the "largeenough" number of hidden units implies the requirement for anabritrarily large amount of training data.  Isn't one of the keystrengths of a deep network not the deep functional architecture perse, but rather its ability to make more efficient use of the availabledata when learning the weights, due to its hierarchical structure?This would certainly be the conclusion I would be tempted to draw fromTable 1, for example, where you demonstrate that a deep networkconsiderably outperforms a shallow network with the same number ofparameters, but that the deep structure isn't actually necessary toproduce equivalently-good decision boundaries.  So my answer to thequestion posed in the second paragraph of the introduction wouldprobably be (g).  There argument that the shallow network couldperform better, if only the regularization were more effective or thelearning algorithm improve, just isn't supported by the experiments.And sadly there is no theoretical justification either.This is a shame, because as I said, the experiements are nice, andthere are lots of future directions the work could go.  In speechrecognition, for example, it would be interesting to investigatewhether the learner/teacher methodology would work across differenttraining sets: for example, for speaker adaptation; or multi-lingualor cross-domain ASR.  There are other interesting questions that areleft unanswered: for example, what would happen if you had a muchlarger quantity of unlabelled data, and a teacher network trained on alabelled subset.  Would the mimic network assymptotically approach theperformance of the teacher, or given the extra data, could the methodexceed the performance of the teacher?  I suspect not, but it would begood to find out.In summary, this paper an interesting concept, but it needs to bepresented quite differently, with more theoretical analysis and lessspeculation.Some specific points:090 -> "training on these logit values makes learning easier for theshallow net by placing emphasis on confident predictions" - really?308 -> "shallow net achieves accuracy comparable to CNNs" - this is a bit misleading as the accuracy is some way off that of the teacher CNN here.


A State-Space Model for Decoding Auditory Attentional Modulation from MEG in a Competing-Speaker Environment
Humans are able to segregate auditory objects in a complex acoustic scene, through an interplay of bottom-up feature extraction and top-down selective attention in the brain. The detailed mechanism underlying this process is largely unknown and the ability to mimic this procedure is an important problem in artificial intelligence and computational neuroscience. We consider the problem of decoding the attentional state of a listener in a competing-speaker environment from magnetoencephalographic (MEG) recordings from the human brain. We develop a behaviorally inspired state-space model to account for the modulation of the MEG with respect to attentional state of the listener. We construct a decoder based on the maximum a posteriori (MAP) estimate of the state parameters via the Expectation-Maximization (EM) algorithm. The resulting decoder is able to track the attentional modulation of the listener with multi-second resolution using only the envelopes of the two speech streams as covariates. We present simulation studies as well as application to real MEG data from two human subjects. Our results reveal that the proposed decoder provides substantial gains in terms of temporal resolution, complexity, and decoding accuracy.

In the last few years, there has been considerable interest in the neuroscience community in decoding high-level latent brain states from noisy neural recordings. One such state variable is attention: primates and other high-level organisms can preferentially distribute resources to encode and process a selective set of incoming stimuli in a way that is typically not externally visible. In the case of auditory selective attention, empirical studies have identified a set of neural variables that can be measured that provide information about which sound a subject is currently directing attention to. Some of these variables can be measured via magnetoencephalography (MEG). This study capitalises on these prior observations to build a statistically-principled decoder of human auditory attentional states under competing-speaker auditory stimulation. The authors explain the underlying measurements, represent the problem as one of Bayesian inference, then present an EM-based inference procedure for inferring the latent variables. The authors demonstrate the performance of this algorithm on simulated data, then test it on real MEG data where subjects were directed to attend in a specified way.At a high level, this is a well-motivated study on a topical problem, and the authors have executed it well.Quality: As far as I can tell, the inference procedure is technically sound. The authors demonstrate its application to synthetic data where ground truth is known, and they are able to recover the ground truth reasonably well. The results on real data are reasonably convincing, insofar as these are very limited and noisy measurements. It would be useful if the authors added a comment on how to relate the SNRs in the synthetic data with the real data, given that the noise in the real MEG signals (Fig 3, top two rows of each panel) are rather high.Clarity: Generally, this is well explained, though there were a few points in the text which could have been better motivated :-    - There's some confusion on spectrotemporal vs temporal features of speech streams in this paper. It is true that some previous work uses spectrotemporal information from the speech stream (e.g. Mesgarani & Chang, Ding and Simon). The model presented in this paper only makes use of temporal information. The authors do make this clear initially. But Line 49 refers to Fig 1A as being about the spectrotemporal information (while the image only shows the temporal information). On Line 71, the authors wonder whether one can decode using a "more parsimonious set of spectrotemporal features", but really they are asking whether they can rely on temporal features alone. On Line 94, the authors claim they will decode the attentional state given the spectrotemporal features of the two speech streams, but only do so in terms of the temporal speech streams.    - DSS: I'm not an expert on this denoising technique, but describing the method as "removing inconsistent temporal components that are not phase-locked to the stimulus" sounds an awful lot like it would suppress markers of attentional shifts (which are not typically stimulus-locked). I suspect that this is just a poor choice of wording.    - Section 2.2.1. It really wasn't clear to me until much later on that this section is describing the likelihood. This is perhaps the most confusing part of the paper. The authors should clarify that that the generative model encapsulated in equation (3) is never inverted directly, but rather tested via the surrogate, theta. The authors do need to explain carefully why they rely on this surrogate measure, and not just, for example, consider $y_{t,r}$ as conditionally Gaussian with mean $e_{i,t}$. My suspicions are either that there are slow gain fluctuations in the neural signal, due to changes in global states such as arousal, as well as stimulus-driven gain changes, which the correlation "detrends", or it introduces a nonlinearity that assists in the temporal smoothing. Whether it is because of these or other reasons, the authors should explain why.    - It was not clear to me why the von Mises (prior/posterior) distributions on theta (e.g. eqn (8)) should be zero-mean/mode. Figure 2B of Ding & Simon 2012 suggest that the correlations (across time) between $e_{i,t}$ and $y_{t,r}$ should be positive for both attended and unattended streams. Originality: This provides a more rigorous and principled approach to auditory attentional-state decoding than the other papers I have seen on this topic (e.g. by Ding & Simon, Mesgarani & Chang), and would likely be a welcome step forward in this domain. Some references would be appreciated for the statement "Previous studies employ time-averaging..." on Line 50. Significance: This kind of decoding has a strong practical component for scientists interested in studying attentional states, and engineers developing tools for interacting with the attending brain. There's clearly a fair way to go on this topic still (the performance of this algorithm is ultimately limited), but it's a good step forward.Some additional comments:- Line 125. If the authors have room, they should insert a paragraph break before "In a competing-speaker environment."- Line 110: "It is known that the TRF is a sparse filter". The TRF is not known, but estimated. For future work, the authors may consider a different regularisation scheme -- I suspect that the TRF is better described as temporally localised. The components of Fig 1C after 250ms look very artefactual, and are unrealistic given the functional characteristics of the auditory system. The authors may wish to consider Park & Pillow's ALD as an alternative method for estimating tau.- Lines 169-171 "The subjective experience... by the listener." This is irrelevant, and should be cut. - Line 416. It's a bit rich to blame the variability in estimated switching time on the subject. It's certainly possible, but it would be more parsimonious to blame the algorithm. If the authors want to comment on this, they should sample from the individual-trial posteriors $p_{k,r}$ to estimate the strength of this prediction. Also, it appears to be that the model's assumption of slowly-varying attentional state (via the AR1 dynamics on $z_k$) breaks down under the instruction that the subject switch attention during the pause. The authors should note this.- I suggest changing the Bernoulli success probability symbol from $p_k$ to $q_k$, to disambiguate from the pdf symbol $p$. Also equation (10) should start with $p$


Spectral Clustering of Graphs with the Bethe Hessian
Spectral clustering is a standard approach to label nodes on a graph by studying the (largest or lowest) eigenvalues of a symmetric real matrix such as e.g. the adjacency or the Laplacian. Recently, it has been argued that using instead a more complicated, non-symmetric and higher dimensional operator, related to the non-backtracking walk on the graph, leads to improved performance in detecting clusters, and even to optimal performance for the stochastic block model. Here, we propose to use instead a simpler object, a symmetric real matrix known as the Bethe Hessian operator, or deformed Laplacian. We show that this approach combines the performances of the non-backtracking operator, thus detecting clusters all the way down to the theoretical limit in the stochastic block model, with the computational, theoretical and memory advantages of real symmetric matrices.
Clustering a graph into groups or functional modules (sometimes called communities) is a central task in many fields ranging from machine learning to biology. A common benchmark for this problem is to consider graphs generated by the stochastic block model (SBM) [7,22]. In this case, one considers n vertices and each of them has a group label g v ∈ {1, . . . , q}. A graph is then created as follows: all edges are generated independently according to a q × q matrix p of probabilities, with Pr[A u,v = 1] = p gu,gv . The group labels are hidden, and the task is to infer them from the knowledge of the graph. The stochastic block model generates graphs that are a generalization of the Erdős-Rényi ensemble where an unknown labeling has been hidden.
We concentrate on the sparse case, where algorithmic challenges appear. In this case p ab is O(1/n), and we denote p ab = c ab /n. For simplicity we concentrate on the most commonly-studied case where groups are equally sized, c ab = c in if a = b and c ab = c out if a = b. Fixing c in > c out is referred to as the assortative case, because vertices from the same group connect with higher probability than with vertices from other groups. c out > c in is called the disassortative case. An important conjecture [4] is that any tractable algorithm will only detect communities if
where c is the average degree. In the case of q = 2 groups, in particular, this has been rigorously proven [15,12] (in this case, one can also prove that no algorithm could detect communities if this condition is not met). An ideal clustering algorithm should have a low computational complexity while being able to perform optimally for the stochastic block model, detecting clusters down to the transition (1).
So far there are two algorithms in the literature able to detect clusters down to the transition (1). One is a message-passing algorithm based on belief-propagation [5,4]. This algorithm, however, needs to be fed with the correct parameters of the stochastic block model to perform well, and its computational complexity scales quadratically with the number of clusters, which is an important practical limitation. To avoid such problems, the most popular non-parametric approaches to clustering are spectral methods, where one classifies vertices according to the eigenvectors of a matrix associated with the network, for instance its adjacency matrix [11,16]. However, while this works remarkably well on regular, or dense enough graphs [2], the standard versions of spectral clustering are suboptimal on graphs generated by the SBM, and in some cases completely fail to detect communities even when other (more complex) algorithms such as belief propagation can do so. Recently, a new class of spectral algorithms based on the use of a non-backtracking walk on the directed edges of the graph has been introduced [9] and argued to be better suited for spectral clustering. In particular, it has been shown to be optimal for graphs generated by the stochastic block model, and able to detect communities even in the sparse case all the way down to the theoretical limit (1).
These results are, however, not entirely satisfactory. First, the use a of a high-dimensional matrix (of dimension 2m -where m is the number of edges -rather than n, the number of nodes) can be expensive, both in terms of computational time and memory. Secondly, linear algebra methods are faster and more efficient for symmetric matrices than non-symmetric ones. The first problem was partially resolved in [9] where an equivalent operator of dimensions 2n was shown to exist. It was still, however, a non-symmetric one and more importantly, the reduction does not extend to weighted graphs, and thus presents a strong limitation.
In this contribution, we provide the best of both worlds: a non-parametric spectral algorithm for clustering with a symmetric n × n, real operator that performs as well as the non-backtracking operator of [9], in the sense that it identifies communities as soon as (1) holds. We show numerically that our approach performs as well as the belief-propagation algorithm, without needing prior knowledge of any parameter, making it the simplest algorithmically among the best-performing clustering methods. This operator is actually not new, and has been known as the Bethe Hessian in the context of statistical physics and machine learning [14,17] or the deformed Laplacian in other fields. However, to the best of our knowledge, it has never been considered in the context of spectral clustering.
The paper is organized as follows. In Sec. 1 we give the expression of the Bethe Hessian operator. We discuss in detail its properties and its connection with both the non-backtracking operator and an Ising spin glass in Sec. 2. In Sec. 3, we study analytically the spectrum in the case of the stochastic block model. Finally, in Sec. 4 we perform numerical tests on both the stochastic block model and on some real networks.

Summary:This paper proposes a new spectral approach for clustering/community detection of graphs.It builds upon a recent approach based on the non-backtracking random walk operator whose spectral properties allow community detection or clustering of graphs generated by the Stochastoc Block Model down to the theorectical limit.Observing that the non-backtracking random walk operator is non-symmetric, high-dimensional and complicated, the authors of this submission propose using a simpler, symmetric operator, the Bethe Hessian, and show that its spectrum is closely related to that of the non-backtracking walk operator and achieves the same theoretical performance limit. The authors also derive connections to Bethe free energy of a certain Ising model, offering perspectives from spin glass theory and phase transition of random graphs.Experiments on synthetic networks confirm their theorectical result. On several real networks where the assumption of Stochastic Block Modelmay not hold, the proposed algorithm still performs quite well.  Quality: This work is of decent quality: the proposed approach is interesting and potentially useful in real applications, the authors provide clear arguments about its performance guarantee and connections to other methods. The only thing that is a bit disappointing is the experiment on real data, where details (e.g., sizes of networks, parameters of algorithms, competitors) are missing. Clarity:The paper is written in a succinct and clear way, but there are a few places that may benefit fromextra explanation: * Lines 185-187: It is easy to see as r gets closer to 1, the Eigenvalues of H become non-negative because H(1) is the usual Laplacian.But the relation between "B's Eigenvalues come in pairs having their product close to \rho(B)" and "for r < r_c, the negative eigenvaluesof H(r_c ) move back into the positive part of the spectrum" is less clear. Can the authors provide a bit more explanation? * Line 189: What is d and what is the notation <>?* Eq. (17): The Jacobian of F in (15) should be a (2M+1) x (2M) matrix, but Eq. (17) seems to be (2M+1) x (2M+1). It looks like removing the first column and replacing the first row of J in (17) with -1's gives the correct Jacobian.* Eigen vectors of Bethe Hessian: Section 2.1 gives connections of the spectrums, but how about the eigenvectors? How do the Eigenvectors of the Bethe Hessian relate to those of the non-backtracking walk operator? How do they reveal community structures? Originality:The proposed approach can be viewed as an elegant reformulation of the non-backtracking walk approach, though the main techinical result facilitating the reformulation, as pointed out by the authors, has been presented in earlier work.Therefore it is not particularly strong in terms of originality.Significance:The proposed approach seems to be a promising addition to the suite of spectral clustering techniques.Avantages over other spectral methods are that the number of negative Eigenvalues provides an estimate of the number of clusters, there is a well-defined way to set the parameter r, making the algorithm tuning-parameter free, and it is guaranteed to detect the communities generated by a Stochastic Block Model down to a theoretical limit.However, whether it should be chosen over other spectral methods on real data is not clear given the experimental results in the paper. The authors stated the performances are comparable to other spectral methods, but did not give details of the methods. For example, how was the number of clusters chosen? Is number of negative Eigenvalues of Bethe Hessian a good estimate of number of communities?Also, there are no timing results to demonstrate the better computational efficiency of the proposed method than the non-backtracking approach. Most of the real networks in the paper are small, which limits the significance of the experimental results.


Learning on graphs using Orthonormal Representation is Statistically Consistent
Existing research [4] suggests that embedding graphs on a unit sphere can be beneficial in learning labels on the vertices of a graph. However the choice of optimal embedding remains an open issue. Orthonormal representation of graphs, a class of embeddings over the unit sphere, was introduced by Lovász [2]. In this paper, we show that there exists orthonormal representations which are statistically consistent over a large class of graphs, including power law and random graphs. This result is achieved by extending the notion of consistency designed in the inductive setting to graph transduction. As part of the analysis, we explicitly derive relationships between the Rademacher complexity measure and structural properties of graphs, such as the chromatic number. We further show the fraction of vertices of a graph G, on n nodes, that need to be labelled for the learning algorithm to be consistent, also known as labelled sample complexity, is Ω where ϑ(G) is the famous Lovász ϑ function of the graph. This, for the first time, relates labelled sample complexity to graph connectivity properties, such as the density of graphs. In the multiview setting, whenever individual views are expressed by a graph, it is a well known heuristic that a convex combination of Laplacians [7] tend to improve accuracy. The analysis presented here easily extends to Multiple graph transduction, and helps develop a sound statistical understanding of the heuristic, previously unavailable.

The paper presents statistical analyses of graph transduction algorithms, based on orthonormal embedding of graphs introduced by [17]. The main results provided in the paper are: 1) providing a deterministic upper bound on transductive Rademecher complexity in terms of a graph-structural measure (Chromatic number/Lovasz-theta function) for a class of graphs.2) using that relation to establish graph-dependent generalization bounds for transduction algorithms -  the high level proof technique for this part follows [10], but several modification for the current setting are done. 3) establishing consistency of a particular LS labeling algorithm, using the above results for several families of graphs. A multiple graph extension is also provided, although it is straight-forward given the previous results. The results provided are interesting - particularly the deterministic connection between Rademacher complexity and the structural properties of the graph. The subsequent results are naturally based on this connection. I wonder if there could be tighter upper bounds based on concentration results for \lambda_1(K), that hold with high probability which involves some interpretable parameter of the graph ? The paper is not well-written which makes following the proofs and interpreting the results time-consuming. Presentation could be improved significantly. I would suggest to move some proofs (or all) in the main paper to the supplementary section and improve the presentation in the main paper (by interpreting  the results). Furthermore, there are several places where the style could be made formal (e.g., 163-164; 212-217). Also in algorithm 1, should it be \hat y = U^T h* ? There are several typographical errors that need to be corrected (e.g., supplementary pdf - line 216 - dictribution -> distribution). Also minor changes like providing the original definition of v(G) and then presenting the alternate identity from [14] (lines 124-125) needs to  be done.


Spectral Methods for Indian Buffet Process Inference
The Indian Buffet Process is a versatile statistical tool for modeling distributions over binary matrices. We provide an efficient spectral algorithm as an alternative to costly Variational Bayes and sampling-based algorithms. We derive a novel tensorial characterization of the moments of the Indian Buffet Process proper and for two of its applications. We give a computationally efficient iterative inference algorithm, concentration of measure bounds, and reconstruction guarantees. Our algorithm provides superior accuracy and cheaper computation than comparable Variational Bayesian approach on a number of reference problems.

Spectral methods are based on decomposing moment tensors.  If data are generated from latent variable models, their empirical moment tensors have a kind of (approximate) low-rank decomposition (approximate due to both noisy observations and finite sample estimation error in the empirical moments).  These decompositions can be computed from the moment tensors, e.g. using a kind of power iteration method on related (symmetrized) tensors.  The basic ideas of these spectral methods are fairly well established and many examples have been explored, especially in [6].This paper applies spectral fitting methods to data modeled as being generated from an IBP, including two common emission models (a linear gaussian model and a sparse factor analysis model, described in Section 2).  The main ingredients are a calculation of the appropriate moment tensors and corresponding symmetrized versions (Section 3), an application of the standard tensor power decomposition method (Section 4), and concentration proofs to offer recovery guarantees when data are generated from the model (Section 5).  There are also experiments (Section 6).The paper applies existing methodology and 'turns the crank' in the context of the IBP. While having such methods available for the IBP is interesting to the community, their development would fit better as an example in a paper like [6] than a full NIPS paper. Therefore the technical contribution and originality seem low.There could be contributions here that I'm not appreciating. For example, there could be substantial deviations from the standard machinery that are required, either for the algorithm or for the analysis in Section 5. In developing the algorithm, there is a need to use fourth-order moments (Section 3.1), though this doesn't seem like a significant leap. In the analysis of Section 5 the authors suggest (around line 281) that they provide a more general methodology for producing relevant concentration of measure bounds, but again it is not made clear whether this is a contribution of great originality or significance outside of the scope of this paper. If there are contributions in this paper that differentiate it from a 'turn the crank' application of standard machinery, the authors should highlight them in their response and clarify them in the text.The paper is decently written overall, though some points that seem relevant are not discussed. For example, Section 5 seems cut short and seems a bit disconnected from the rest of the paper. It could benefit substantially from more explicit connections back to the IBP. In addition, it could be made clear how the notion of concentration of measure interacts with the nonparametric generative model; does the size of K grow with the number of samples m? (The references [6,7] do not mention the Dirichlet process explicitly as far as I can tell, so I'm not sure if the nonparametric aspect of the story is made clear in some existing work without a more direct reference.) Also, does the notation for the number of samples switch between m and n in Sections 5.1 and 5.2?The experiments in Section 6 seem a bit weak. Both the first and second involve data that are essentially generated from the model (the first being synthetic data and the second being the IBP classic of objects-in-images) and therefore they don't shed much light on whether the method can perform well when data are not extremely close to following the model's assumptions, a question which seems especially relevant for spectral methods. In the short discussion of the third experiment, there isn't much to reveal the advantages and disadvantages of the proposed spectral algorithm over other algorithms. If the proposed method and analysis are essentially an application of existing machinery to the IBP, then experiments could provide a real contribution in shedding light on when spectral methods may be appropriate and when they may fail. But this experiments section does not provide such insight.Here are some miscellaneous points:- Since [6,7] don't mention the Dirichlet process explicitly (based on searching the text), the reference around line 38 seems strange.- Line 113 says all proofs are deferred to the appendix, but one proof remains in the paper.- The {0;1} on line 134 should probably be {0,1}, though that sentence could be revised for clarity.- The comment on line 142 that "[t]his is undesirable for the power method" could be clarified.- The plots in Figure 1 look like they were saved to a raster format, which makes them hard to read. The text and line sizes should probably be adjusted.- In Figure 3, was the MCMC algorithm stopped short at 1.72 seconds, or was there some reason to believe it was stuck in such a suboptimal mode?


Analysis of Learning from Positive and Unlabeled Data
Learning a classifier from positive and unlabeled data is an important class of classification problems that are conceivable in many practical applications. In this paper, we first show that this problem can be solved by cost-sensitive learning between positive and unlabeled data. We then show that convex surrogate loss functions such as the hinge loss may lead to a wrong classification boundary due to an intrinsic bias, but the problem can be avoided by using non-convex loss functions such as the ramp loss. We next analyze the excess risk when the class prior is estimated from data, and show that the classification accuracy is not sensitive to class prior estimation if the unlabeled data is dominated by the positive data (this is naturally satisfied in inlier-based outlier detection because inliers are dominant in the unlabeled dataset). Finally, we provide generalization error bounds and show that, for an equal number of labeled and unlabeled samples, the generalization error of learning only from positive and unlabeled samples is no worse than 2 √ 2 times the fully supervised case. These theoretical findings are also validated through experiments.

The paper presents an analysis of learning for a binary classification task when only positive and unlabeled data are available. As the authors point out this setting can be found in many applications, in particular in outlier detection. The authors begin by showing that PU classification can be modelled as a cost-sensitive classification problem. It is also argued that in such settings a non-convex loss is necessary, the argument here is based on the shortcomings of the hinge-loss specifically with respects to the ramp loss. In the modelling proposed by the authors the minimized risk depends on an estimate of the class prior. As in practice one rarely has access to the true prior, the authors introduce the notion of effective prior, which is shown to directly influence the risk. They further show that at least in the cases where the true class prior is large the effective prior is not influenced by small variations in the estimated prior. The authors also provide generalization error bounds in the case of kernel machines. They argue that the expected 0-1 loss in the case of PU classification is only 2*sqrt(2) times that of the fully supervised setting. Finally the paper presents empirical evidence in favour of the ramp loss over the hinge loss. The paper is overall well written and the subject is I think quite interesting. I am not personally aware of similar work, though I have not worked on PU classification. Multiple facets of the problem are theoretically analyzed, and I think in this respect the paper is not lacking. However there are a number of issues that prevent me from recommending acceptance. I would be interested on hearing from the authors on these points.- In section 3, I believe the authors overstate their findings. They state (lines 193), that to obtain the correct decision boundary, the loss function should be symmetric and thus the use of a non-convex loss is unavoidable. I don't see how the necessity for symmetry has been proven. In my view all that has been shown is that the ramp loss does not incorporate this "superfluous" term. Furthermore non-convexity does not necessarily follow from symmetry. As an aside, I think it is interesting to note the connection between the hinge loss terms in this settings and the Universum.     - In section 4, line 248 states that the risk when applying \hat{f} is on a line tangent to R* at \pi=\hat{\pi}and that the difference of \hat{\pi} from true \pi results in the excess risk. However in figure 3, the line is tangent at \pi=\tilde{\pi}. Also, the notation here is not helpful as \pi is used to denote the true prior and a variable.- In section 5, the authors state that the generalization error bound is at most 2*sqrt(2) times the error bound in the supervised case. I don't see why this holds, it seems to me that the authors are arguing \frac{2\sqrt{2}}{\sqrt{n + n'}} \geq  \frac{1}{\sqrt{n'}}  + \frac{1}{\sqrt{n}}, but this is not true. In any case, I don't statements of this sort don't make sense when working with big-O classes, as constants are suppressed. Also, in the abstract this relationship is mistakenly stated to relate to the errors, when in fact it relates to the bounds.- The experimental results are not entirely convincing. To be sure the ramp loss dominates the hinge loss as expected from the analysis. However results are shown only on a single dataset and on a small subset of the binary tasks that can be derived (only those involving zero). I understand that the main contribution is theoretical but it does take away a bit. What is truly missing however is how \hat{\pi} was estimated in these experiments. Was the true prior used?- A perhaps minor issue, but the notation in section 2 is not clean. P_1 is used for two different distribution P(X|Y=1) and P(f(X)=1|Y=1).


Minimax-optimal Inference from Partial Rankings
This paper studies the problem of rank aggregation under the Plackett-Luce model. The goal is to infer a global ranking and related scores of the items, based on partial rankings provided by multiple users over multiple subsets of items. A question of particular interest is how to optimally assign items to users for ranking and how many item assignments are needed to achieve a target estimation error. Without any assumptions on how the items are assigned to users, we derive an oracle lower bound and the Cramér-Rao lower bound of the estimation error. We prove an upper bound on the estimation error achieved by the maximum likelihood estimator, and show that both the upper bound and the Cramér-Rao lower bound inversely depend on the spectral gap of the Laplacian of an appropriately defined comparison graph. Since random comparison graphs are known to have large spectral gaps, this suggests the use of random assignments when we have the control. Precisely, the matching oracle lower bound and the upper bound on the estimation error imply that the maximum likelihood estimator together with a random assignment is minimax-optimal up to a logarithmic factor. We further analyze a popular rankbreaking scheme that decompose partial rankings into pairwise comparisons. We show that even if one applies the mismatched maximum likelihood estimator that assumes independence (on pairwise comparisons that are now dependent due to rank-breaking), minimax optimal performance is still achieved up to a logarithmic factor.

This work studies the use of the Plackett-Luce model on the problem of rank aggregation with partial feedbacks. It mainly consists in a theoretical study of the estimation error depending on the quantity of feedbacks that are given by the users and depending on the assignments of the items to the users. The authors provide first two lower bounds on the estimation error  of the PL model – one for any estimator, the other (tighter) only for unbiased estimators. Then, the authors present three different upper bounds, depending on the how the feedback is observed (partial ranking or pairwise preferences), with two schemes for the observations of the pairs.I liked the technical quality of this theoretical analysis (even if I didn’t checked the proof deeply) for several reasons:-	The results proved are really non-trivial,-	the authors paid attention to enlarge the scope of their results when they could – e.g. the first lower bound holds for a larger family of models than PL,-	the authors only narrowed the scope to obtain tighter bound and actually provided some insights to compare their different bounds.-	The discussion is not restricted on the pure performance of the estimation w.r.t. the quantity of feedback, but also with the way the feedback are assigned (even if I’m not familiar with this setting).Beyond these aspects, truth should be said, the major drawback of this paper concerns itsform. The writing and the structure (1 and ½ page of notations and definitions at the beginning) all contribute to make it extremely difficult to read.


Distributed Parameter Estimation in Probabilistic Graphical Models
This paper presents foundational theoretical results on distributed parameter estimation for undirected probabilistic graphical models. It introduces a general condition on composite likelihood decompositions of these models which guarantees the global consistency of distributed estimators, provided the local estimators are consistent.

Paper Summary:--------------The paper focuses on the problem of distributed parameter estimation in MRFs through the use of a composite likelihood function. The authors identify a class of composite likelihood functions for which one can learn the parameters of an MRF in a fully-parallel but globally consistent manner. The authors introduce the Strong LAP Condition characterizing the graph structure of an MRF and show that if the auxiliary MRF A_q corresponding to a clique for interest q satisfies the Strong LAP condition then globally consistent estimates of the parameters corresponding to A_q can be computed locally. This new condition on the structure of the MRF allows the authors to tie together two prior approaches by Liu et al, and Mizrahi et al. for distributed parameter estimation. Moreover, the conditional LAP formulation extends the original LAP algorithm to densely connected graphs. High-level comments:-------------------The paper studies the important problem of distributed parameter estimation in PGMs. I think the techniques put forward in this paper provide an interesting theoretical basis for characterizing families of networks that are amenable to efficient distributed learning and inference algorithms with consistency guarantees. The technical development is sound and the corresponding definitions, prepositions and theorems are clearly stated. The paper presents some novel ideas as it extends prior work on LAP. More precisely, prior work was using marginal distributions to estimate network parameters, thus, evaluating the partition function, while the current framework shows how parameter estimates can be derived using conditional distributions, and hence, avoiding the computation of the partition function. Finally, the proposed framework unifies two different models previously proposed. However, the discussion on the differences between LAP and Strong LAP should be improved. Detailed comments on this can be found in the weak points below. Detailed comments:-------------------1) Strong points:(a) Interesting formulations and sound theoretical analysis. (b) Unifying framework over two different lines of work. To some extend this work completes is a complement to previous work by Liu et al. by providing the necessary conditions under which the assumptions of Liu et al. hold.(c) The theoretical framework was used to derive efficient localized parameter estimators. It will be very interesting how these techniques apply to real data, and how they compare with LAP under instances where LAP is also efficient. 2) Weak points:(a) The comparison between LAP and strong LAP should be improved. Now, Theorem 9 and Proposition 2 seem to state the same thing. Notice that the way they are stated Theorem 9 includes Proposition 2, i.e., normalized potentials and identifiable parameters. So either Theorem 9 or Proposition 2 should be restated as there is no difference between them. In fact theorem 9 seems to be trying to draw a connection between preposition 8 and the previously stated LAP condition but this is done without success. (b)The discussion in the intro should be extended to give a high-level description of why the Strong LAP condition leads to a more efficient estimator. In fact the reason is the computation of conditional versus marginal distributions. Also, section 5 should be extended with examples. Figure 1 is there but no mention of it is included in section 5.(c) The statement about the complexity of LAP being exponential in the number of cliques (the same for the centralized estimators) should be that it is exponential in the number of network variables since computing the partition function enumerates all possible assignments of values to these variables. So basically exponential in the clique size but not the number of cliques.  (d) Typos: Some minor typos Page 1, line 034 allows us transfer -> allows us to tranfer Page 1, line 039 lead -> led Page 1, line 040 factorization - > factorization Page 2, line 086 i.e. -> i.e.,  Page 3, line 160 i.e. that -> i.e., (without that) Page 3, line 143 optimising - > optimizing Page 3, line 149 optimisation -> optimization


Learning From Weakly Supervised Data by The Expectation Loss SVM (e-SVM) algorithm
In many situations we have some measurement of confidence on "positiveness" for a binary label. The "positiveness" is a continuous value whose range is a bounded interval. It quantifies the affiliation of each training data to the positive class. We propose a novel learning algorithm called expectation loss SVM (e-SVM) that is devoted to the problems where only the "positiveness" instead of a binary label of each training sample is available. Our e-SVM algorithm can also be readily extended to learn segment classifiers under weak supervision where the exact positiveness value of each training example is unobserved. In experiments, we show that the e-SVM algorithm can effectively address the segment proposal classification task under both strong supervision (e.g. the pixel-level annotations are available) and the weak supervision (e.g. only bounding-box annotations are available), and outperforms the alternative approaches. Besides, we further validate this method on two major tasks of computer vision: semantic segmentation and object detection. Our method achieves the state-of-the-art object detection performance on PASCAL VOC 2007 dataset.

Summary: The authors present a scheme for semantic labeling and object detection. The former is accomplished by generating candidate segments and extracting features for these; and then regressing on the Intersection-over-Union overlap ratio between a candidate segment and pixel-accurate ground truth. In the case of object detection, only weaker labels in terms of bounding boxes are available and the precise spatial extent is used as latent variable. Regression is performed using a weighted SVM and a weighted transductive SVM, respectively. Results are on par with the state of the art in semantic labeling, and the best currently known for object detection on VOC 2007.Originality:Algorithmic novelty is limited, and the authors unfortunately do not cite pertinent previous work. In particular, the weighted SVM is well-known (e.g. [R1,R2,R4] below, not cited). The authors put a sum constraint on the weights, which may be novel (I did not conduct an extensive literature search here). The "latent e-SVM" that the authors propose is a weighted version of a transductive SVM (e.g. [R3], not cited). [R1] Suykens et al. Weighted least squares support vector machines: robustness and sparse approximation, Neurocomputing, 2002[R2] X.Yang et al. Weighted Support Vector Machine for Data Classification, IJCNN, 2005[R3] V. Vapnik, â€œStatistical Learning Theoryâ€, 1998[R4] M.Lapin et al, Learning Using Privileged Information: SVM+ and Weighted SVM, Neural Networks, 2014Significance:Outperforming the recent CNN results by a "conventional" learning approach is a feat and of great interest to the community. Quality:Crucial references to previous work are missing (see above). The object detection framework involves quite a bit of engineering, but not outside the norm. The default method for regression on the unit interval is (regularized) logistic regression, so these results should be reported as baseline to support the use of a sum-constrained weighted SVM. Authors should give more details on how they picked regularization coefficients \lambda_W and \lambda_R. The main strength of the paper are the good empirical results. Clarity:The title and abstract do not summarize the core contribution of the paper well. The abstract also insinuates that regression targets from the unit interval are somehow less informative than mere binary class memberships. This is not true, because the latter can always be obtained from the former by thresholding. For a NIPS audience, the paper spends rather too much space (up to and including page 4) on small variations of well-known algorithms, and passes somewhat quickly over the all-important feature engineering details (lines 254ff).  Minor comments               Line 19: continues => continuousLine 94: boxes => boxLine 98: AP: Abbreviation not introducedLine 350: SVM => SVCLine 350: with every different values => for all threshold values-----I have upped my quality score by one, assuming the authors will not be defensive about the relation to previous work, but rather comment extensively on it and connections to it.


Learning Mixed Multinomial Logit Model from Ordinal Data
Motivated by generating personalized recommendations using ordinal (or preference) data, we study the question of learning a mixture of MultiNomial Logit (MNL) model, a parameterized class of distributions over permutations, from partial ordinal or preference data (e.g. pair-wise comparisons). Despite its long standing importance across disciplines including social choice, operations research and revenue management, little is known about this question. In case of single MNL models (no mixture), computationally and statistically tractable learning from pair-wise comparisons is feasible. However, even learning mixture with two MNL components is infeasible in general. Given this state of affairs, we seek conditions under which it is feasible to learn the mixture model in both computationally and statistically efficient manner. We present a sufficient condition as well as an efficient algorithm for learning mixed MNL models from partial preferences/comparisons data. In particular, a mixture of r MNL components over n objects can be learnt using samples whose size scales polynomially in n and r (concretely, r 3.5 n 3 (log n)

This paper studies a rank data model as a mixture of Luce (or Plackett-Luce) Model. They provide conditions for estimating parameters of this model along with a moment-based algorithm. They further relate their theory to the algorithm.This paper has nice technical results and they are presented well. I have a couple of concerns/suggestions with the following pieces of the presentation:1-    The authors state that they are learning Mixture Multinomial Logit models. Despite being closely related, their model is only considering Mixture of Luce model. The concerning differences are as follows:      i) MNL originally and traditionally, is used when you also observe alternative characteristics (also named as Multinomial logit regression) and the significance of research on Mixed MNL and MNL are due to the econometrics application. However, this paper does not get into that aspect and does not relate the results to an econometric setting.      ii) In most of applications, MNL is used for choice data, where we observe top 1 choice of agents (in fact their market shares). However, the theories and algorithms in this paper do not talk about this case. In fact, this paper only concerns pairwise data either observed directly as independent pairs or after the pre-process from full rankings. Therefore, I suggest clarifying the beam of this paper and its implications in MNL and MMNL. 2-    Since theorem 3 guarantees the correctness with high probability, isn’t the correctness of the algorithm an approximate correctness? Also, a condition (ASSUMPTION 1) in David Hunter’s paper (MM algorithms for generalized Bradley-Terry models. In The Annals of Statistics , volume 32, pages 384–406, 2004) depicts that there are cases that can lead to irreducibility of the defined graph, with the data. From Hunter et al.: "As noted by Ford (1957), if it is possible to partition the set of individuals [alternatives] into two groups A and B such that there are never any inter group comparisons, then there is no basis for rating any individual in A with respect to any individual in B. On the other hand, if all the intergroup comparisons are won by an individual from the same group, say group A, then if all parameters belonging to A are doubled and the resulting vector renormalized, the likelihood must increase; thus, the likelihood has no maximizer. The following assumption [Ford (1957)] eliminates these possibilities." And I wonder if Hunter's assumption 1 is not needed for the correctness of the proposed algorithms?3-     I can’t make sense of proposition 2.14-     There are no experiments comparing the proposed algorithm with former works and there are no experiments describing the performance which is claimed theoretically. There is a large body of work and algorithms for Mixed Multinomial Logit model (as the authors indicated in introduction) and I really liked to see a comparison with some real data or a synthetic setting.The problem and results of the paper are presented clearly.1-    As the authors indicate, there is a lot of work done in this line of research and there are some papers which algorithmically address the exact same estimation problem as this paper and in terms of application it is not clear what the gain of the new algorithm is. For example, the following papers proposed algorithms for estimating mixture of placket-Luce Model:A mixture of experts model for rank data with applications in election studies Authors: Isobel Claire Gormley, Thomas Brendan MurphyClustering ranked preference data using sociodemographic covariatesAuthors: Isobel Claire Gormley, Thomas Brendan MurphyGeneralized Random Utility Models with Multiple Types Authors: Hossein A. Soufiani, Hansheng Diao , Zhenyu Lai, David C. Parkes,A Markov Chain Approximation to Choice Modeling Authors: Jose Blanchet, Guillermo Gallego, Vineet Goyal2-    The results on identifying conditions for the mixture of PL models is original and interesting. Authors mention their results as “identifiable” in the discussion. However, it is not clear whether this is a formal statistical identifiablity claim.3-    Combining moment-based methods for ranks in order to find efficient estimators is interesting, and have been used in the following paper: Azari et al., Computing Parametric Ranking Models via Rank-Breaking and same authors, Generalized Method-of-Moments for Rank Aggregation. It is interesting to see how this work relates and compares to their methods experimentally and theoretically.The interest in this work can be two folds, from an operations research aspect or rank aggregation aspect.From the operation research point of view, this work lacks clear connection between the claims and an application through computational experiments to illustrate the implications of the new model for the specific OR problem.From the rank aggregation aspect, and viewing the model as mixture of PL or BTL, the conditions and results are interesting, and with clear presentation this is an original contribution.


Spike Frequency Adaptation Implements Anticipative Tracking in Continuous Attractor Neural Networks
To extract motion information, the brain needs to compensate for time delays that are ubiquitous in neural signal transmission and processing. Here we propose a simple yet effective mechanism to implement anticipative tracking in neural systems. The proposed mechanism utilizes the property of spike-frequency adaptation (SFA), a feature widely observed in neuronal responses. We employ continuous attractor neural networks (CANNs) as the model to describe the tracking behaviors in neural systems. Incorporating SFA, a CANN exhibits intrinsic mobility, manifested by the ability of the CANN to support self-sustained travelling waves. In tracking a moving stimulus, the interplay between the external drive and the intrinsic mobility of the network determines the tracking performance. Interestingly, we find that the regime of anticipation effectively coincides with the regime where the intrinsic speed of the travelling wave exceeds that of the external drive. Depending on the SFA amplitudes, the network can achieve either perfect tracking, with zero-lag to the input, or perfect anticipative tracking, with a constant leading time to the input. Our model successfully reproduces experimentally observed anticipative tracking behaviors, and sheds light on our understanding of how the brain processes motion information in a timely manner.

Summary:The authors present a continuous attractor neural model which implements (anticipative) tracking. The authors show that spike frequency adaptation (SFA) can induce traveling waves under certain conditions. Interestingly, they show that the effects induced by SFA are similar to those that can be obtained by introducing asymmetric coupling between neurons as in [14], with the advantage that this method does not depend on hard-wired connections. The bulk of the paper is a theoretical analysis of a simplified model and simulations with the complete model for verification. The model can sustain traveling waves and reproduces both perfect tracking and anticipatory tracking. Comments to the authors:The paper is well written and interesting - especially the link between the model of [14] and the demonstration that SFA reproduces a similar effect.  I am not an expert in this field, so I find it hard to judge how novel this work is - for instance, you refer to [12], saying that it was already known that a continuous attractor network with SFA can exhibit traveling waves, but in the abstract you state that you propose the SFA mechanism to implement tracking. I think it would be good to clearly delineate what your new contributions are in the introduction.Are the time constants used in the model (e.g. at figure 2B) realistic? Could you comment on this? A factor 60 difference between \tau and \tau_v seems large to me.How well does the model reproduce experimental results? Such comparison would make the paper more interesting.Minor issues:l. 53. unpredictable flash: I don't think this has to do with the predictability of the flash - it can flash at regular intervals. The point is that the flashing stimulus does not move. l. 53-55. Despite ...; rephrase -> e.g., Although it is clear that the brain ..., it remains unclear ... .l. 59-60. moving speed of an object -> speed of a moving objectl. 81. anticipative time -> anticipative timesl 100, 2 ~ 5 ms \rho is -> 2 ~ 5 ms, \rho is the neural density, and l 150, dynamical property -> dynamical propertiesl 151, of adding the -> of addingfig 2 B, units of v_int?l 208, traveling wave -> traveling wavesl 215, the simulation -> simulationl 215, that at -> that inl 216, the Gaussian shape -> Gaussian shapel. 232, is dominated -> are dominatedl. 258, effect on -> effect ofl. 304-306, which gives to (twice) -> gives rise to?l. 320-321, SFA holds these appealing -> exhibits these properties.l. 354, Prefect -> Perfectl. 360, the condition that ... -> the approximationl. 361, We carry -> We carriedl. 265, of stimulus speed -> of stimulus speeds throughout the Ms.: holding a traveling wave -> contains a traveling wave / can exhibit traveling waves?


Discovering, Learning and Exploiting Relevance
In this paper we consider the problem of learning online what is the information to consider when making sequential decisions. We formalize this as a contextual multi-armed bandit problem where a high dimensional (D-dimensional) context vector arrives to a learner which needs to select an action to maximize its expected reward at each time step. Each dimension of the context vector is called a type. We assume that there exists an unknown relation between actions and types, called the relevance relation, such that the reward of an action only depends on the contexts of the relevant types. When the relation is a function, i.e., the reward of an action only depends on the context of a single type, and the expected reward of an action is Lipschitz continuous in the context of its relevant type, we propose an algorithm that achievesÕ(T γ ) regret with a high probability, where γ = 2/(1 + √ 2). Our algorithm achieves this by learning the unknown relevance relation, whereas prior contextual bandit algorithms that do not exploit the existence of a relevance relation will haveÕ(T (D+1)/(D+2) ) regret. Our algorithm alternates between exploring and exploiting, it does not require reward observations in exploitations, and it guarantees with a high probability that actions with suboptimality greater than are never selected in exploitations. Our proposed method can be applied to a variety of learning applications including medical diagnosis, recommender systems, popularity prediction from social networks, network security etc., where at each instance of time vast amounts of different types of information are available to the decision maker, but the effect of an action depends only on a single type.

The paper deals with the problem of contextual bandits in the following setting: The context is composed of the product of D continuous sets, which are throughout considered (w.l.o.g) to be the interval [0,1]. The gain of an arm is assumed to be a random variable with mean equal to some Lipschitz function that depends on exactly one of the D intervals. The contexts are assumed to be arbitrary. In each round the player chooses an arm and gains its defined gain. The player may choose to query the value of that gain at a cost. Eventually, the objective is to minimize the regret defined as the sum of gains of the optimal strategy minus the sum of gains earned by the player plus the sum of exploration costs.The novelty w.r.t the setting is the sparsity of the function, meaning the fact that it depends on only one interval rather than all of them. This additional assumption is very natural and has been considered in other scenarios as well. Having it allows a significant increase in the performance of the algorithm. Specifically, it allows the authors to obtain a regret of T^c with constant c that is independent of the context dimension D (without the assumption there is a lower bound of c \geq (D+1)/(D+2)). In order to deal with the continuous regime, the authors use a (somewhat) standard technique of maintaining partitions of the interval with adaptively shrinking lengths. To deal with the sparsity of the gain function, the algorithm maintains, for each arm, estimates of the function for pairs of intervals originating from different intervals and determines the relevant interval (i.e., the interval that determines the value of the arm) by finding the one whose expected value is the same for all pairs containing it. Overall, the techniques are mostly technical but are definitely non-trivial.The authors provide experiments in a setting where algorithm is used to combine several different classifiers. These experiments do not do justice with the main contribution of the paper since the dimension of the context is chosen to be 3. I would expect the experiments to compare the current algorithm with those that do not assume sparsity, and an advantage would only occur for large values of D. In particular, for D=3, previous results provide a regret of T^{0.8} while this paper guarantees a regret of T^{0.83}.Comments:•	line 253: D should be |D|•	Line 361: The paper should state the exact dependence on D_{rel}•	The following paper could be relevant: Agarwal, Alekh, et al. "Taming the Monster: A Fast and Simple Algorithm for Contextual Bandits.". In particular, it would be interesting to see whether the mentioned setting can be solved using their techniques as well as the resulting regret.•	Appendix, line 103: log(tAD/\delta) should be log(t^4AD/\delta)•	Appendix, line 161, missing L


Improved Distributed Principal Component Analysis
We study the distributed computing setting in which there are multiple servers, each holding a set of points, who wish to compute functions on the union of their point sets. A key task in this setting is Principal Component Analysis (PCA), in which the servers would like to compute a low dimensional subspace capturing as much of the variance of the union of their point sets as possible. Given a procedure for approximate PCA, one can use it to approximately solve problems such as k-means clustering and low rank approximation. The essential properties of an approximate distributed PCA algorithm are its communication cost and computational efficiency for a given desired accuracy in downstream applications. We give new algorithms and analyses for distributed PCA which lead to improved communication and computational costs for k-means clustering and related problems. Our empirical study on real world data shows a speedup of orders of magnitude, preserving communication with only a negligible degradation in solution quality. Some of these techniques we develop, such as a general transformation from a constant success probability subspace embedding to a high success probability subspace embedding with a dimension and sparsity independent of the success probability, may be of independent interest.

This paper considers the problem of trading off the communication and computation cost of distributed computation and proposes a new distributed k L-2 error fitting algorithm. The proposed algorithm can be seen as a combination of many previous speed up techniques for distributed PCA and clustering methods. However, the authors also contribute optimizations over the base methods and further improves the communication and computation efficiency. The theoretical guarantee is sound and experiments are convincing.What is the subroutine of A_\alpha in Algorithm 1? Moreover, algorithm 2 involves algorithm 1 which uses A_\alpha. However, it seems in Theorem 6 that both the communication and computation cost is independent of \alpha? Could the authors provide some explanations?Line 275, how shall we understand that for the subspace embedding, there is \|HAy\|_2 = (1\pm \epsilon)\|Ay\|_2?In Theorem 6, the failure probability depends on s and t. I suppose t is equal to t1 and t2, which also depends on s. Could the authors optimize the probability and remove the dependence on t? As for the current statement, suppose t1=t2=O(\log s), then the probability is 1-O(1/s+1). Is this result good enough? Could the authors provide the explicit form for the constant c_0? Such that the readers can better understand the tightness of the bound.My biggest concern is about the performance measure. As mentioned in line 85, it is desired to find a center set \mathcal{L}’ such that the relative error is small, d^2(P, \mathcal{L}’ ) \leq (1+\epsilon) \min_{\mathcal{L}}d^2(P,\mathcal{L}). It is expected to see a similar relative error bound for algorithm 2 in Theorem 6. In the experiments, the authors should provide the value of parameter \epsilon, or the values of t1 and t2. Minor comments:Line 93, I suggest to use “reducing communication cost” as the paragraph title, which more clearly expressed the contributions compared with “improved communication”. Similarly, line 108, “improved computation” should also be revised.


On Model Parallelization and Scheduling Strategies for Distributed Machine Learning
Distributed machine learning has typically been approached from a data parallel perspective, where big data are partitioned to multiple workers and an algorithm is executed concurrently over different data subsets under various synchronization schemes to ensure speed-up and/or correctness. A sibling problem that has received relatively less attention is how to ensure efficient and correct model parallel execution of ML algorithms, where parameters of an ML program are partitioned to different workers and undergone concurrent iterative updates. We argue that model and data parallelisms impose rather different challenges for system design, algorithmic adjustment, and theoretical analysis. In this paper, we develop a system for model-parallelism, STRADS, that provides a programming abstraction for scheduling parameter updates by discovering and leveraging changing structural properties of ML programs. STRADS enables a flexible tradeoff between scheduling efficiency and fidelity to intrinsic dependencies within the models, and improves memory efficiency of distributed ML. We demonstrate the efficacy of model-parallel algorithms implemented on STRADS versus popular implementations for topic modeling, matrix factorization, and Lasso.

The authors proposed a framework of primitives for model parallelism, in order to address the problem that a single machine may be inadequate to fit a big model in memory.This seems to be an interesting paper, well written and organized, albeit some small typos are found (e.g., q_idx -> idx in Figure 4, 1/L -> 1/P in line 140).Toward the significance of this paper, I have the following concerns:1. I agree that for models such as deep neural networks (e.g. CNN), more parameters often leads to better performance, as demonstrated empirically by [5] (using the paper's bib reference number), which justifies the need of more efficient model parallelism framework for bigger models. However, for models like topic models (LDA) and matrix factorization (MF), I am skeptical toward such need. For example, would LDA with more than tens of thousands of topics, or MF with rank larger than a few thousands, lead to better model performance, compared against their counterparts with moderate number of parameters? The work could be more interesting if the authors address this issue. Besides, can the proposed framework be applied to models that already proved to enjoy more parameters, such as deep neural networks? If yes, how would it perform when compared to the one proposed in [5]? In short, I think more compelling use cases for the proposed framework would greatly enhance the potential impact of this work on the NIPS community.2. The experiments.1) Some important specifications for the experiments are missing. On the evaluation metrics for LDA and MF, how was the convergence being measured? Take LDA, was the log-likelihood calculated on training documents or testing (unseen) documents? One would imagine LDA with huge number of topics can easily over-fit, if no special regularization or sparsity-inducing priors being adopted, and such convergence could be meaningless. And how was the memcached (which is one of the keys for the scalability for YahooLDA) configured? And for the proposed framework, how were the summary statistics D and B stored and distributed to the work nodes, as their size grow proportionally to the number of parameters? These are just some key factors that may need to be clarified, as they may contribute to the difference observed in Figure 8 and 9.2) The comparison between STRADS and Graphlab in the MF experiment is somehow unfair. Because STRADS used the cyclic coordinate descent (CCD) as its optimization algorithm, whereas Graphlab used alternating least square (ALS). As it has been demonstrated in [21], CCD converges faster than ALS and is also more scalable. As a result, it would be more interesting to compare STRADS against Graphlab with the *same* optimization algorithm. Furthermore, the method proposed in [9] is related, and would also be an interesting baseline for comparison. With such additional experimental results, one may gain further insight into the STRADS framework.


A Representation Theory for Ranking Functions
This paper presents a representation theory for permutation-valued functions, which in their general form can also be called listwise ranking functions. Pointwise ranking functions assign a score to each object independently, without taking into account the other objects under consideration; whereas listwise loss functions evaluate the set of scores assigned to all objects as a whole. In many supervised learning to rank tasks, it might be of interest to use listwise ranking functions instead; in particular, the Bayes Optimal ranking functions might themselves be listwise, especially if the loss function is listwise. A key caveat to using listwise ranking functions has been the lack of an appropriate representation theory for such functions. We show that a natural symmetricity assumption that we call exchangeability allows us to explicitly characterize the set of such exchangeable listwise ranking functions. Our analysis draws from the theories of tensor analysis, functional analysis and De Finetti theorems. We also present experiments using a novel reranking method motivated by our representation theory.

SUMMARYThe paper derives a representation theorem for the class of "exchangeable" listwise scoring functions employed in learning to rank problems. The results show that for finite instance spaces, such functions are expressible as a particular combination of similarity functions over pairs of instances. For compact spaces, a similar result holds for the case of lists of size two; for general lists, the result holds under additional assumptions by appeal to deFinetti's representation theorem.SIGNIFICANCEThe issue of designing listwise loss functions has been well studied, but the design of listwise scoring functions less so. As noted in the paper, some works focus (e.g. [4]) involve scoring each instance within the list with a single function, albeit one that is optimised in a listwise sense. The results in the paper are interesting, and while largely intuitive  not a-priori obvious. The stated goal is to give a recipe with which to guide the design of listwise ranking functions. I think the observation that this can be done by designing similarity functions on pairs of instances, and suitably combine them, is a good first step towards this goal.All results rely on a symmetry assumption on the listwise ranking function. This seems reasonable, and minimally holds e.g. for the pointwise case. The results for the compact case in 3.2.2 additionally rely on the scoring function being an unnormalised probability density over the instances. It is less clear to me how reasonable this is. The requirement of nonnegativity of the scoring function could be handled by an appropriate link function (e.g. exp(.) as done in the experiments). But this would require suitable interplay with the surrogate loss being optimised to ensure convexity, something that may deserve some comment.The experiments show that it is possible to post-process the outputs of a pointwise ranker (even one trained to optimise a listwise loss) using the representation theorem. The experiments are the weakest part of the paper, and likely more could be done to make a convincing case for the value of the recipe (e.g. try a wider class of pairwise functions, have a comparison of pairwise "as-is" versus pointwise to check the perils of overfitting, et cetera). But good performance in this regime is still interesting, as (to my knowledge) improving upon methods trained to optimise listwise losses is non-trivial. So I think they are indicative of the promise of the representation theorem in guiding the design of scoring functions.QUESTIONS/COMMENTS- The statement of Theorem C.1 apparently holds for m > 2 instances in the list. This seems to not agree with the text; perhaps the point is that existence of a multilinear symmetric decomposition would imply the result? If so the appeal to Theorem 3.8 in the proof seems misleading. It does seem like you could have such a corollary to 3.11, so perhaps that should be stated.- Even if a simple consequence of 3.10, I think a proof of Theorem 3.11 should be included in the appendix. The statement of the theorem alone leaves some questions:	* There is an implicit "for every m" needed? In which case, f should actually be defined for every possible m in X^m?		* In 3.10 the "g" function is (I presume) the marginal, derived appropriately from the joint p. Does a similar restriction exist here in terms of the function g?	* The requirement of an infinite sequence in deFinetti must of course carry over to the theorem, but it is a little hard to understand what exactly it is requiring of the ranking function and/or the input space. The comment about the result requiring "infinitely many objects for which every leading finite subsequence is exchangeable" could be expanded. Isn't this implicit when one assumes that f is symmetric in its arguments, and hence exchangeable regardless of the provided inputs (which could be one of infinitely many possible values, since X is infinite)?	* What does it mean for theta to be a "random distribution" in the prelude to the theorem?- The extension of 3.11 to the partially symmetric case seems non-obvious, and minimally requires a careful statement of the infinite sequence assumption, and the appropriate boundedness of the integral of f. I would recommend making this explicit.OTHER COMMENTS- I like that the paper builds up from the simple case of finite instance spaces to a more general theory.- Line 211, for three documents f_i : X^3 \to R, so f_i is a tensor? Or do you group the last two elements into one?- Remark II pg 5, tensor completion seems reasonable for finite spaces and suitably large training sets, but in practice it seems likely that one would face cold-start problems (no observations for a particular "row") due to limited coverage of the instance space.- Line 268, is the corresponding decomposition theorem referred to Theorem C.1?- It might be better to include Theorem C.1 in the body and defer 3.8 to the appendix if space is an issue.- Appendix F is unclear. What are r, s? How exactly is the surrogate loss used for optimisation?- The paper is generally well written. There were some small typos worth addressing:	* Def 2.1, \to instead of \mapsto in definition of f_j.	* Missing period line 169.	* Period instead of comma line 186.	* Theorem 3.8 proof, missing boldface in x, y.	* Theorem 3.11, we need f : X^m \to R_+?	* Fix capitalisation in ref [3].- Other comments:	* Why the use of braces in { x_{\i} }?	* Include summation indices in Prop 3.5, 3.6, Thm 3.8?	* Def 3.4, use \pi rather than \sigma for consistency?	* Theorem C.1 proof should come after 3.8.	* Notation clash in (14), S for training set and S_k for similarity function?


A Synaptical Story of Persistent Activity with Graded Lifetime in a Neural System
Persistent activity refers to the phenomenon that cortical neurons keep firing even after the stimulus triggering the initial neuronal responses is moved. Persistent activity is widely believed to be the substrate for a neural system retaining a memory trace of the stimulus information. In a conventional view, persistent activity is regarded as an attractor of the network dynamics, but it faces a challenge of how to be closed properly. Here, in contrast to the view of attractor, we consider that the stimulus information is encoded in a marginally unstable state of the network which decays very slowly and exhibits persistent firing for a prolonged duration. We propose a simple yet effective mechanism to achieve this goal, which utilizes the property of short-term plasticity (STP) of neuronal synapses. STP has two forms, short-term depression (STD) and short-term facilitation (STF), which have opposite effects on retaining neuronal responses. We find that by properly combining STF and STD, a neural system can hold persistent activity of graded lifetime, and that persistent activity fades away naturally without relying on an external drive. The implications of these results on neural information representation are discussed.

Summary: The authors present a model of persistent activity in neural networks as typically observed in working memory tasks and modeled as attractor dynamics. The authors note the shortcomings of existing models, namely the reliance on implausible global excitatory or inhibitory signals to reset the network dynamics after settling into an attractor state and the superfluousness of such a stable attractor, since in working memory tasks, the dynamics need only persistent as long as the task requires, not indefinitely. In the proposed model, these issues are confronted by incorporating short term synaptic facilitation and depression into a network model. Using a mean-field approach, the authors identify stable fixed points of the rate dynamics and how these fixed-points change as a function of network connectivity and timescale parameters. Using this formalism, the authors identify a family of parameters for which the network exhibits only 1 true stable fixed point- corresponding to network quiescence- but also exhibits a second, "slow point", i.e. a point which under different network parameters would correspond to a 2nd unstable fixed point of the rate dynamics, but for the chosen parameters yields network dynamics that evolve extremely slowly towards the true, stable fixed point. The primary effect of this “slow point” is to create a quasi-stable firing rate, similar to the attractor dynamics of existing models, but different in that the dynamics will eventually collapse to the quiescent state. The authors illustrate the existence of this state over a range of timescales of the synaptic plasticity and instantiate the presented framework in a full, spiking neural network.Review: This is an exceptionally clear-written submission of great interest to the theoretical neuroscience community. It stands out compared to alternative work in attractor dynamics in that it assigns a useful computation role for short-term synaptic plasticity in generating these dynamics and for noting that real biological systems have limited use for “true” fixed-points. The authors eloquently note that neural dynamics need only remain stable for finite, not infinite time, and propose a biologically plausible mechanism to both generate and terminate this quasi-stability. The reduced, mean-field model is clearly presented and illustrates quite nicely the relationship between network parameters and the stability of the slow dynamics. All the figures positively contribute to the submission and create a simple visual representation of the text’s mathematical details. The authors do a great job of explaining the opposing roles facilitation and depression play in generating the slow network dynamics, as most clearly displayed in Figure 4. All-around a solid submission, however, two small points for improvement. 1. The authors claim that the spiking model exhibits irregular, independent activity (critical to invoke a mean-field assumption)  which is difficult to see in Figure 5a, and upon closer inspection, appears to be untrue. As best as I can tell, the spike trains appear extremely regular, and even begin to synchronize a bit (as can be seen in the firing rate, around 2.75 seconds in Figure 5B). The main analytical results seem to hold true as presented here, but rely on this assumption. Clearly displaying the irregularity and independence in the spiking simulations would improve the submission. 2. The main analytical results are derived under the assumption that the synaptic plasticity variables have reached a steady-state, despite the fact that, as the authors note, these variables evolve slowly compared to the rate. A discussion of the conditions under which this assumption is valid would be helpful.