Unsupervised Transcription of Piano Music
We present a new probabilistic model for transcribing piano music from audio to a symbolic form. Our model reflects the process by which discrete musical events give rise to acoustic signals that are then superimposed to produce the observed data. As a result, the inference procedure for our model naturally resolves the source separation problem introduced by the the piano's polyphony. In order to adapt to the properties of a new instrument or acoustic environment being transcribed, we learn recording-specific spectral profiles and temporal envelopes in an unsupervised fashion. Our system outperforms the best published approaches on a standard piano transcription task, achieving a 10.6% relative gain in note onset F 1 on real piano audio.

The paper focuses on the important problem of polyphonic transcription in music information analysis/retrieval. The paper suggest a novel generative model with inspiration form piano music; however, the model might be generalized to other instruments, or maybe multiple instrument tracks. An (unsupervised) inference algorithm is developed, and the method is compared to state-of-the-art methods on the MAPS corpus data. Superior performance is demonstrated both in onset and frame-level evaluation metrics. Despite issues mentioned in detailed comments below, the paper is a solid and significant contribution that might be of interest to the NIPS community.Quality:The paper is relevant and well-written and describe in good detail a novel generative piano music model and arguments for most of the involved model ingredients (some issues mentioned in detailed comments). The model is sufficiently linked and contrasted to existing state-of-art techniques. The methods benefits from being unsupervised and has some generalizability to new instruments although this has only been tested in a rather limited setting. The performance as regards onset and frame-level metrics is superior to state-of-art techniques and the error patterns is argued to bee musically plausible despite the fact that such issues have not been directly been addressed in the model. Whether this is a unique feature of the proposed framework (over alternative methods) is, however, not discussed. The overall conclusion is that the paper is a significant contribution to the community and might inspire further development. Issues and needed clarifications are addressed in the detailed comments below.Clarity: • The paper is well-written and clearly formulated except issues addressed below. It is not possible to do exact reproduction of the results without more details - in particular as regards the experiments.• The references are relevant and adequate.               Significance:• The technical contribution seems sufficiently interesting to a part of the NIPS community related to audio/music, and to sequential models in general. The modeling framework is not complete as the adaption to novel un-seen instruments is not explicitly addressed (except implicit general nature noise models). Further, musical constraints which might help producing perceptually acceptable errors is not addressed. Finally only a single but feasible inference scheme is suggested, hence, the potential benefits of more advanced inference methods is not elicited.Originality:• The overall modeling framework is original and tries to include more aspects of the generation of the oberserved music spectrum. Detailed comments: • Line 033: test data > training data• Last part of intro lines 087-091 should go to the conclusion section.• Line 104 (and other places): loudness is a subjective/perceptual quality. Better to use amplitude/ intensity / strength.• Line 181: activation will remain low? Do you mean low or zero?• Line 185: Gaussian noise distribution. What is this stochastic variable going to model? How do you handle potential negative activations?• Sec 2.3: What is the noise in the spectrum going to represent?• Sec 2.3: Do you have any evidence for the claim that Poisson is the best noise model? Maybe refer to an available technical report.• How is the priors on alpha and sigma going to be constructed, and how important/sensitive is the method to such choices? Have you considered music genre/style variations?• Iterated conditional mode is the chosen as theinference method. Have you considered alternatives, and what are the arguments for sufficiency in the present case.• Sec 3.1. Some details on inferring M is missing. Provide a complete description.• How are the hyper-parameters beta, gamma and xi specified?• The experimental section is rather brief. For reproducibility reasons it would be relevant to refer to a technical report on the details.• The difference between ENSTDkAm and ENSTDkC1 is ambient or close-in recording of the Yamaha Diskklavier. Hence they might be rather similar! Please make clear that the generalizability to a bigger variety of pianos is unclear.• Why is 23ms (actually 23,22ms) a good window size?• Why is 50ms used in evaluation of onsets. 50ms corresponds to 1200 bpm and notes played above this rate might perceptually sound as a single tone. That could be part of the explanation.• Explain in more detail the music possible errors in line 402. Do the alternative methods have same desirable property?• Lines 418-420: Observed errors are claimed not to affect the re-synthesized audio. What is the evidence? How could you take such tolerable error patterns into consideration in the modeling framework?• The generalizability of the model to other instruments or maybe polyphonic music as such is not addressed. Please include some discussion.• Some references seem to be incomplete and ordering of text e.g. 'In Proceedings' should be reversed.


Near-Optimal-Sample Estimators for Spherical Gaussian Mixtures
Many important distributions are high dimensional, and often they can be modeled as Gaussian mixtures. We derive the first sample-efficient polynomial-time estimator for high-dimensional spherical Gaussian mixtures. Based on intuitive spectral reasoning, it approximates mixtures of k spherical Gaussians in d-dimensions to within 1 distance using O(dk
computation time. Conversely, we show that any estimator requires Ω dk 2 samples, hence the algorithm's sample complexity is nearly optimal in the dimension. The implied time-complexity factor O k, is exponential in k, but much smaller than previously known. We also construct a simple estimator for one-dimensional Gaussian mixtures that usesÕ(k 2 ) samples andÕ((k ) 3k+1 ) computation time.

This paper considers PAC learning of mixture spherical Gaussian distributions. Via exploiting the spherical structure of the data (in particular, the assumption that all covariance matrices of all components are diagonal), the authors are able to derive efficient algorithms to estimate the means and the standard deviations. The mixtures can then be derived using a modified Scheffe's method, This paper is, in general, very well written. The intuition, the proposed method, and the theoretical justifications are all stated clearly and reader-friendly. I can follow the main contents very easily. The related literature seems to be well cited (I am not an expert in this area, so not very confirmative). The results look novel, correct, and important. Overall, I enjoy reading it a lot. However, there are still some issues need to be addressed:(1) It looks like that the whole paper is focused on the setting that k, the number of components, is known. It is a little bit surprising that the authors did not discuss the situation when k is unknown. Is it trivial to estimate it? At least some heuristics are preferred. (2) By assuming sphericity on the components, The results look like quite restrictive in real applications. Some discussions on retionality of sphericity assumption, or a generalization to more general cases, are desired. (3) Technically, the bound on delta, provided in line 288, is a little bit weird. For example, if d is fixed, this delta can be as large as 1/3. Minor comments:(1) Line 47-48, rephrase the sentence.(2) line 247, S looks more like a covariance-type matrix.(3) line 270, "necessary" -> "necessarily"?


Feedforward Learning of Mixture Models
We develop a biologically-plausible learning rule that provably converges to the class means of general mixture models. This rule generalizes the classical BCM neural rule within a tensor framework, substantially increasing the generality of the learning problem it solves. It achieves this by incorporating triplets of samples from the mixtures, which provides a novel information processing interpretation to spike-timing-dependent plasticity. We provide both proofs of convergence, and a close fit to experimental data on STDP.

Summary:The paper extends the classical BCM learning rule to utilize information from spike triplets.It is shown that the update rule can learn selectivity for mixture distributions (by convergingto the class means). Quality:By employing tensor notation, the paper shows that the BCM rule can be generalized to use information frommore than a pair of spikes (spike triplets are used for the examples). While the model has fewer parametersthan previous learning algorithms based on spike triplets or quadruplets and the method can be shown to havestable points as class means of mixture distributions, the lack of experimental comparisons with other modelsmakes it hard to gauge the incremental contribution of the model.Clarity:The paper is in general easy to follow but the derivation of equations on line 269 and 273 was unclear to me. Originality:STDP utilizing spike triplets has been introduced in the past by Pfister and Gerstner. Also, Froemke and Danpublished a model which uses spike quadruplets. Moreover, it seems to me that under Poisson spiking statisticsand certain choice of parameters, the model from Pfister and Gerstner is equivalent to the model described here.In that light, I believe a clear comparison should be made using simulations of the algorithm with those fromPfister/Gertsner, Froemke, etc. The simulations should clearly demonstrate under what circumstances does thepresented model differ (in the expected sense) and is better than that from Pfister and Gerstner.Significance:The paper extends a classical approach in neural learning (BCM) and demonstrates that the extended approach has broadercapabilities than the original approach. However, clear and detailed quantitative comparison to previous research in the field is missing.[1] Pfister, Jean-Pascal, and Wulfram Gerstner. "Triplets of spikes in a model of spike timing-dependent plasticity." The Journal of neuroscience 26.38 (2006): 9673-9682.Notes:Line 79, Baysian -> BayesianLine 99 \delta_{ij} is the more common notation rather than \delta_i^jLine 106 In the definition of c, subscript i should be added.		 Does the example here have multiple neurons or a single neuron and dimensional data? 		 If the former, m_i, m_{i-1} and d_i should be bold in the equaution. If not, c_i needs		 to be defined as m_{i-1} x d_i (without inner product and bold letters) to clarify its 		 a single neuron.Line 147, 148 I think there should be an expectation in front RHS equations on both linesLine 161 m should be bold in R(m)Line 262 same as aboveLine 351 Missing verbLine 410 "Lower figure" needs to be corrected. The figures are next to each other.


Predictive Entropy Search for Efficient Global Optimization of Black-box Functions
We propose a novel information-theoretic approach for Bayesian optimization called Predictive Entropy Search (PES). At each iteration, PES selects the next evaluation point that maximizes the expected information gained with respect to the global maximum. PES codifies this intractable acquisition function in terms of the expected reduction in the differential entropy of the predictive distribution. This reformulation allows PES to obtain approximations that are both more accurate and efficient than other alternatives such as Entropy Search (ES). Furthermore, PES can easily perform a fully Bayesian treatment of the model hyperparameters while ES cannot. We evaluate PES in both synthetic and realworld applications, including optimization problems in machine learning, finance, biotechnology, and robotics. We show that the increased accuracy of PES leads to significant gains in optimization performance.

This manuscript proposes a new acquisition function for use in Bayesian optimization with a Gaussian process belief on a latent objective function.  The proposed criterion is a rewriting of the previously suggested entropy search (ES) objective [Hennig and Schuler, JMLR 2012], which seeks to maximize the posterior entropy of the _location_ of the global maximum x*, H[x* | D].  We sequentially greedily select the next point x' in the domain of the objective which maximizes the expected posterior entropy H[x* | D'], where D' is the current data augmented by the proposed new observation (x', y').  This requires marginalizing the unknown observation value y' and, if we are to be fully Bayesian, the hyperparameters of the model, \theta.  The authors first write out the mutual information between x* and y' (given D) in the way that it would most often be encountered in an active learning context (and how entropy search was originally posed by Hennig and Schuler).  The expression is unwieldy due to the marginalization of y' and the repeated need to approximate H[x* | D'], which requires many approximations.The authors instead propose to use a trick that has been previously used in other similar contexts, e.g., [Houlsby, et al. NIPS 2012]: they rewrite the mutual information so that only _marginal_ entropies of y' are required.  These are often much easier to work with.  The new objective is H[y' | x', D] - E[ H[y' | x', D, x*] ], where the expectation is taken over x*.  The former is simply the predictive entropy of a GP, which can be computed in closed form assuming fixed hyperparameters.  The second term is more novel and interesting; we must approximate the predictive uncertainty of y', conditioning the GP on the location of the global maximizer x*.  Conditioning a GP on x* is intriguing, and the bulk of the paper is about an approximate method for doing so.  First, the authors propose a method for sampling from the current distribution p(x* | D).  The proposed method for doing so exploits a now-common trick of approximating the covariance from samples in the Fourier domain.  This gives each sampled function a finite representation, allowing us to find its maximum easily (although the authors omit details on this point).  Given a location x*, they must condition the GP on knowledge of its location.  This would be rather difficult (it essentially requires a censored observation at every point in the domain), so the authors instead propose a set of relaxed but reasonable requirements:* x* must at least be a local maximum.  This is easy to accomplish in the GP framework by conditioning (analytically) on observations forcing the gradient to be zero and the Hessian to be negative definite.  The latter could be difficult in high dimension, so we instead condition H[f] at x* to be diagonal with negative entries.* The value at x* must exceed all those previously seen.* The observed value y' must be less extreme than the value y* we selected subject to the previous constraint.  This reduces the global maximizer condition to just apply at the point x' under consideration, effectively truncating the predictive distribution p(y' | x', D, x*) at y*.The first condition can be analytically handled in the GP framework, but the truncation required by the second is not tractable, so the authors rely on the well-known expectation propagation algorithm for approximating the joint distribution between y* and the diagonal of the Hessian at x*.  Finally, the authors handle uncertainty in the hyperparameters by marginalizing the acquisition function over \theta; this is done by simply slice sampling from p(\theta | D).  A series of experiments indicate that the proposed method (called "predictive" entropy search, but it's really just a new approximation to the same entropy search method) works well.  In particular, Figure 1 is quite convincing at showing that the PES approximation is much closer to the true objective (approximated by rejection sampling) than the ES approximation.  The experiments are well designed and explained, and code will be released upon publication.  Although PES appears to be usually better than ES, it still loses to simpler methods like expected improvement on some problems.One philosophical issue I have with entropy search is that it's not entirely clear to me that H[x* | D] is somehow a "better" objective than, say, EI.  The scenario I always picture is one where we have determined that the optimum of a function must lie in a low, quite flat, but quite broad valley.  In this case we can be very unsure about the exact location of x*, but can be quite certain about its corresponding value y*.  Do I really always need to locate x* with perfect accuracy if I can guarantee that I can find a point with x with f(x) within \epsilon of y* = f(x*)?  Note that a score such as EI can indicate clearly to me that I expect with high certainty not to find a better point and therefore can stop now.  Further, a lookahead version of EI is in fact the Bayesian optimal policy to the optimization problem with simple regret; see [Gruenewaelder, et al. AISTATS 2010].  [Osborne, et al. LION 2009] used a two-step lookahead version of EI that, although prohibitively expensive, encouraged more exploration than the standard, myopic EI criterion and worked better in practice.  I know that this criticism of entropy search is not unique to me, but I have yet to see someone take it head-on and offer a frank discussion on the point.Overall, I like the paper and think the rewriting of the ES objective, together with the interesting and reasonable approximations made by the authors to derive the PES variant, deserve to be published and discussed further.  I also look forward to an eventual code release.Some minor notes:- There are many stacked parens in this paper that are exactly the same height; for an example, see the extreme right-hand-side of (2).  It would help readability (at least for me) if some judicious use of \big, \Big, etc. were used.- Page 1 line 35 (1/35): You say that the derivatives of f are unavailable, but this does not strictly need to be the case.  Indeed you rely on the ease with which we can condition a GP on derivative observations later.  I could imagine a scenario where it might be useful to condition a function on a derivative observation, perhaps by a finite-different approximation as in [Osborne, et al. LION 2009].- 1/52: A comma after "decision" would make this more clear.- 2/80: maxima -> maximum- 6/306: I don't know where the location (0.572, 0.687) is!- 8/426: "hyperparameters" has not been hyphenated until now.


Consistency of weighted majority votes
We revisit from a statistical learning perspective the classical decision-theoretic problem of weighted expert voting. In particular, we examine the consistency (both asymptotic and finitary) of the optimal Nitzan-Paroush weighted majority and related rules. In the case of known expert competence levels, we give sharp error estimates for the optimal rule. When the competence levels are unknown, they must be empirically estimated. We provide frequentist and Bayesian analyses for this situation. Some of our proof techniques are non-standard and may be of independent interest. The bounds we derive are nearly optimal, and several challenging open problems are posed.

This paper analyzes the Nitzan-Paroush strategy for assigning weights to conditionally (on the outcome) independent experts, which is optimal in expectation. The authors provide exponential concentration inequalities (both upper and lower bounds on probabilities) for the error of this optimal rule. The upper bounds are sharp and derived based on the Kearns-Saul inequality. They continue to analyze the situation where the quality of the experts is not given and must be estimated from data, giving frequentist procedures (which are analyzed) and Bayesian procedures (which are not analyzed). QUALITYThe results are interesting,  nontrivial and appear correct, as far as I checked. I was struck with the use of the Kearns-Saul inequality, which (although I have seen it before and know it was designed for other goals) seems almost magically suitable for application in this problem. The exponential bound Theorem 1(i) is quite strong, being linear in the nr of experts who are correct with probability > 1/2 + epsilon, for fixed epsilon > 0. I do have some questions/small issues about the estimators, both the frequentist and the Bayesian ones. First, as a very minor point, I'd rather call 'adaptive' 'fully empirical'  and 'nonadaptive' 'oracle' by the way, that seems to more correctly describe the difference.Second:Theorem 7: adaptive, high-confidence, frequentist case: This theorem is only useful if the authors can show that the event R will hold with large probability for sufficiently large sample sizes. (Otherwise it may be that we always have to refrain from making a high-confidence decision, making the results useless).The authors should say some more about the sample sizes when we can expect R to hold, given the (oracle, nonadaptive) probabilities(so that 'if we are lucky about the real probabilities, then with high probability we get a situation in which the adaptive bound is useful') About the Bayesian method: this seems to be based on using the standard, unconditional Naive Bayes model in which the experts opinions are viewed as X-data and the outcome as Y. But I guess that most Bayesians would condition on the X's and use conditional likelihood - and then Naive Bayes becomes logistic regression, which often works better anyway. The authors should say something about this additional possibility. Is there any reason why they have not considered it? Why naive Bayes and not logistic regression approach?CLARITYThe paper is quite well-written; I am not an expert in the probabilistic weighted-majority analysis but had no trouble following the paper. ORIGINALITYThe results are somewhat original, the *proof techniques* are very original.  SIGNIFICANCEReasonably high, esp. given the use of the Kearns-Saul inequality.


A Probabilistic Framework for Multimodal Retrieval using Integrative Indian Buffet Process
We propose a multimodal retrieval procedure based on latent feature models. The procedure consists of a Bayesian nonparametric framework for learning underlying semantically meaningful abstract features in a multimodal dataset, a probabilistic retrieval model that allows cross-modal queries and an extension model for relevance feedback. Experiments on two multimodal datasets, PASCAL-Sentence and SUN-Attribute, demonstrate the effectiveness of the proposed retrieval procedure in comparison to the state-of-the-art algorithms for learning binary codes.

This paper defines a joint generative model of an image and its annotated text which is used to learn a bit vector representation for large scale image retrieval.  An Indian Buffet Process is used to learn the length of the bit vector. The method is compared favourably to several widely used techniques. Quality=======It is good to see a retrieval paper constructed around a well-defined probabilistic model.  The model is relatively simple - the assumption that X can be well approximated by ZA seems quite a strong one - it would be nice to see more discussion of the limitations imposed by this assumption. Given that Z is shared, unmodified, between the textual and visual halves of the model, it's not clear how well it would handle images with heavily occluded objects or text that doesn't mention some area of the scene. In addition, the use of a linear-Gaussian model means that the model is unimodal in visual space - which doesn't seem like a good idea. In general, there's quite a lot of discussion of the IBP which could be dropped in favour of more discussion of the model, since that is the main contribution of this paper.The experiments are thorough and show that this approach adds value over other competing approaches.  It would be nice to see more discussion about the reversal of ranking between Fig 3b and 3c.Clarity=======The paper is well written and clearly presented. It would be helpful to include a brief summary of the Euclidean neighbor ground truth labels from [6], since I found it necessary to read that paper in order to understand the differences in relative performance of the proposed method in Fig 3b and Fig 3c.Fig 2. would be made clearer by adding a plate over the images.TyposLine 038: "data-independent"->"data dependent"Line 112: "priori"->"a priori"Line 191: "As the images"->"As for the images"Line 271: "scientic"->"scientific"Originality & significance==========================The main contribution of this paper is the use of a new non-parametric Bayesian model for large scale image information retrieval that gives good performance compared to other commonly used methods.  This is a much more rigorous treatment of the problem than previous more ad-hoc, algorithmic approaches.  It also opens the doors to developing more sophisticated models in this space.  The relevance feedback model also seems like a nice contribution.


Global Sensitivity Analysis for MAP Inference in Graphical Models
We study the sensitivity of a MAP configuration of a discrete probabilistic graphical model with respect to perturbations of its parameters. These perturbations are global, in the sense that simultaneous perturbations of all the parameters (or any chosen subset of them) are allowed. Our main contribution is an exact algorithm that can check whether the MAP configuration is robust with respect to given perturbations. Its complexity is essentially the same as that of obtaining the MAP configuration itself, so it can be promptly used with minimal effort. We use our algorithm to identify the largest global perturbation that does not induce a change in the MAP configuration, and we successfully apply this robustness measure in two practical scenarios: the prediction of facial action units with posed images and the classification of multiple real public data sets. A strong correlation between the proposed robustness measure and accuracy is verified in both scenarios.

The paper gives a nice introduction to sensitivity analysis in graphical models, and proposes a method to check if the MAP configuration will not change, if a simultaneous perturbation in the parameters occurs. The method works by selecting perturbed factors in such a way (using local computations) to create a worst-case scenario. The second best MAP estimation is then identified for the worst case scenario. Theoretically, the complexity is therefore equivalent to the MAP query (exponential in the tree-width), if the number of variables is a constant. As presented, one disadvantage of this technique is that, practically, it requires computational time: n*[MAP inference time], where n is the number of variables. For non-trivial graphical models, the proposed sensitivity analysis can be orders of magnitude slower than the already hard MAP query. That said, more efficient algorithms for getting the M-Best MAP seem to exist and the authors may consider to utilize one of them as a black box. See for example: An Eﬃcient Message-Passing Algorithm for the M-Best MAP Problem, by Dhruv Batra. See also: Search Algorithms for m Best Solutions for Graphical Models, by Rina Dechter et al. On the other hand, the experimental results in the paper do not provide very useful information, as discussed below.Quality:The paper appears to be technically correct. Theorem 1 is trivial and could be a lemma for Theorem 2.  The proof of Theorem 2 in the supplementary material could be re-written in a clearer way to provide more insight. For example, you could discuss the choice of factors in the new PGM as creating a worst-case, and briefly mention that the partition function cancels out and has no effect.On the other hand, the experimental results show that the robustness measure is a good indication for the testing accuracy. While I see why there is a relation between robustness and testing accuracy, I think the relation is much more involved and may be linked to the bias-variance trade-off. Thus, the experiments seem to be distantly related to the current work and not the best choice, or perhaps not well-motivated. I have to admit though that it might be hard to design very meaningful experimental results in this case. The authors could perhaps verify the proof experimentally and show how pessimistic the analysis is, in real-world applications, by having an experimental measure of sensitivity.Clarity:The paper is clearly written when it talks about preliminaries, and related work. When it comes to the contribution, it lacks insightful explanation. Originality:To the best of my knowledge, the contribution is new.Significance:The paper tackles an important problem. Most of the technical results in the paper do not appear to be so surprising though. Worst-case analysis is widely used in many fields. If a system passes the worse case, it passes all cases.  While the paper proposed a novel exact method to do such analysis, in the context of MAP estimation, the inefficiency of the method, and the lack of meaningful experimental results have negatively affected the quality of the work.On the other hand, in some cases, the proposed analysis may be too pessimistic. It might be interesting to try to give a probabilistic guarantee that is more efficient to compute.Other comments/suggestions:In Figure 1 (a): The font is too small.In Figure 1 (b): I suggest adding labels for the axes, and on the plot (maximum, minimum, ...etc).Equation 5: Did you forget to mention that x(1) is the best MAP?UPDATE: *******After reading the authors' response, I am convinced that part of my criticism may have been a bit philosophical, and, therefore, should not lead to rejecting the paper. I am glad to raise my score, and hope the authors can get their final version in the best form, if the paper is accepted.


Deconvolution of High Dimensional Mixtures via Boosting, with Application to Diffusion-Weighted MRI of Human Brain
Diffusion-weighted magnetic resonance imaging (DWI) and fiber tractography are the only methods to measure the structure of the white matter in the living human brain. The diffusion signal has been modelled as the combined contribution from many individual fascicles of nerve fibers passing through each location in the white matter. Typically, this is done via basis pursuit, but estimation of the exact directions is limited due to discretization [1,2]. The difficulties inherent in modeling DWI data are shared by many other problems involving fitting non-parametric mixture models. Ekanadaham et al. [3]  proposed an approach, continuous basis pursuit, to overcome discretization error in the 1-dimensional case (e.g., spikesorting). Here, we propose a more general algorithm that fits mixture models of any dimensionality without discretization. Our algorithm uses the principles of L2-boost [4], together with refitting of the weights and pruning of the parameters. The addition of these steps to L2-boost both accelerates the algorithm and assures its accuracy. We refer to the resulting algorithm as elastic basis pursuit, or EBP, since it expands and contracts the active set of kernels as needed. We show that in contrast to existing approaches to fitting mixtures, our boosting framework (1) enables the selection of the optimal bias-variance tradeoff along the solution path, and (2) scales with high-dimensional problems. In simulations of DWI, we find that EBP yields better parameter estimates than a non-negative least squares (NNLS) approach, or the standard model used in DWI, the tensor model, which serves as the basis for diffusion tensor imaging (DTI) [5]. We demonstrate the utility of the method in DWI data acquired in parts of the brain containing crossings of multiple fascicles of nerve fibers.

This paper proposes elastic basis pursuit, which is an algorithm for fitting sparse mixture models via convex optimisation.  The method combines boosting with an active set method for continuous basis pursuit.  The authors provide proofs of convergence of the algorithm and test it experimentally within the application of estimating fibre directions from DWI data.The algorithm is well motivated, and the authors clearly understand well the wider space of solutions to this family of problem and their pros and cons. It is also well constructed and combines the benefits of different existing approaches retaining computational efficiency, convexity, and accuracy of solution. I am not intimately familiar with the latest literature on this problem, but as far as I am aware, the algorithm is novel and represents a fairly minor but not insignificant advance over the current state of the art.  I have not checked the details of the proofs of convergence, but, assuming they are correct, their presence adds to the value of the paper.The chosen application to demonstrate the technique is timely and appropriate, as diffusion MRI is used widely for estimating microstructural features of brain tissue, including fibre orientations, which then leads to algorithms that reconstruct global brain connectivity and are widely used in neuroscience and neurology. Resolving the orientations of crossing fibres is a long standing problem, which saw a lot of attention from 2000-2010, and still has outstanding issues so improved solutions have significant value.  However, the experimental results presented here are very limited.  They show one simulated example, for which the proposed algorithm will clearly perform better than the others tested, as well as a largely qualitative result comparing fitting error maps over a small brain region.  The simulation results would be much stronger if they included a quantitative evaluation of how much better and how often the new algorithm does in comparison to NNLS and other algorithms used commonly in the diffusion MRI literature.  The brain-data results show no obvious difference between the NNLS and EBP output, although presumably there are significant differences. Maps of number of fibre directions or subsequent tractography would add a great deal to convince the reader that the new fitting algorithm really makes a difference worth thinking about. Even a simple test of the significance of the difference in error scores between NNLS and EBP would add something.Minor points:page 6.  The diffusion tensor in 3D is rank 3 in general.page 7. The form of the oracle function is not clear.page 7.  Should w1, w2, w3 sum to 1?  Or does the value of c ensure the sum of compartment fractions sums to 1? Either way, where is the constraint imposed?  Also, it is unclear how c represents an isotropic diffusion component.page 8. Figures 4 and 5 not referred to from text; fig 4 erroneously referred to as fig 3, but fig 5 never mentioned.


The Infinite Mixture of Infinite Gaussian Mixtures
Dirichlet process mixture of Gaussians (DPMG) has been used in the literature for clustering and density estimation problems. However, many real-world data exhibit cluster distributions that cannot be captured by a single Gaussian. Modeling such data sets by DPMG creates several extraneous clusters even when clusters are relatively well-defined. Herein, we present the infinite mixture of infinite Gaussian mixtures (I 2 GMM) for more flexible modeling of data sets with skewed and multi-modal cluster distributions. Instead of using a single Gaussian for each cluster as in the standard DPMG model, the generative model of I 2 GMM uses a single DPMG for each cluster. The individual DPMGs are linked together through centering of their base distributions at the atoms of a higher level DP prior. Inference is performed by a collapsed Gibbs sampler that also enables partial parallelization. Experimental results on several artificial and real-world data sets suggest the proposed I 2 GMM model can predict clusters more accurately than existing variational Bayes and Gibbs sampler versions of DPMG.

The authors present a novel non-parametric Bayesian model for unsupervised clustering. The model uses a two level hierarchy of Dirichlet process priors to handle clusters which may be multi-modal, skewed and/or heavy tailed. The authors present a collapsed Gibbs sampler for inference which exploits the conjugacy of the model.The authors do an excellent job of motivating the model by explaining the deficiencies of the standard infinite mixture of Gaussians. The review of previous works is good and the model is clearly presented and explained. The model is novel, though conceptually quite similar to the hierarchical Dirichlet process. The authors have chosen an excellent selection of synthetic and real world datasets to benchmark their method. The performance of the method seems to be quite good, in particular the FlowCAP results seem competitive with methods which were tailored to the problem domain.The authors only compare the model against the infinite Gaussians mixture model, with inference done using Gibbs sampling or VB. The paper could be improved if the authors compared against a richer set of models. In particular the authors may want to consider infinite mixtures of student-t's and skew normals to better clarify the advantages their model provides versus distributions which are more flexible than a single Gaussian emission. In addition other probabilistic models for clustering such as the one presented by Iwata et al. [1] are closer to the state of the art for clustering datasets with complex emission distributions.Another minor concern is that authors fix the concentration parameter for the Dirichlet process priors (line 308). I would expect that the model would be somewhat sensitive to the values of these parameters, particularly the relative values at the two levels. It would be helpful if the authors could explore performance on the synthetic dataset across a range of values to determine robustness to these parameters. In addition the method in [2] could readily be applied to place priors on these values instead of fixing them.Another very minor improvement would be for the authors to add details of the distributions and parameterisations they are using in the supplemental material. It would also be good to include explicit equations for the parameter updates described in lines 234-236.[1] Tomoharu Iwata, David Duvenaud, and Zoubin Ghahramani. Warped mixtures for nonparametric cluster shapes. In 29th Conference on Uncertainty in Artificial Intelligence, Bellevue, Washington, July 2013.[2] West, Mike. Hyperparameter estimation in Dirichlet process mixture models. Duke University, 1992.


Beyond Disagreement-based Agnostic Active Learning
We study agnostic active learning, where the goal is to learn a classifier in a prespecified hypothesis class interactively with as few label queries as possible, while making no assumptions on the true function generating the labels. The main algorithm for this problem is disagreement-based active learning, which has a high label requirement. Thus a major challenge is to find an algorithm which achieves better label complexity, is consistent in an agnostic setting, and applies to general classification problems. In this paper, we provide such an algorithm. Our solution is based on two novel contributions; first, a reduction from consistent active learning to confidence-rated prediction with guaranteed error, and second, a novel confidence-rated predictor.

This work is on general sample complexity analysis of active learning.  The paper proposes a strategy that requests labels in a subset of the region of disagreement, and thereby obtains sample complexityguarantees that are sometimes better than those obtained via the disagreement coefficient analysis of disagreement-based active learning methods.  This essentially represents a generalization of the margin-based active learning technique of BBZ07.  The sample complexity bounds are general (applicable to any VC class), and are instantiated for linear separators under distributional assumptions as an example.  Bounds are derived for several noise models, including the realizable case, agnostic case with a given noise rate, and Tsybakov noise.More abstractly, the paper establishes an elegant connection between the work on margin-based active learning techniques as refinements of disagreement-based methods, and the more-recent work of El-Yaniv & Wiener discussing the use of selective classification methods for selective sampling.Specifically, El-Yaniv & Wiener have studied the use of perfect selective classification methods to do active learning in the realizable case, and argued that the use of an optimal such method results in the classic CAL active learning algorithm,which is disagreement-based.  From this perspective, the present work is instead investigating the use of *imperfect* selective classification methods (which have a nonzero, but controlled, probability of error on their predictions), and argues that this approach can recover essentially similar behavior to the existing margin-based methods for linear separators (as a special case of this general approach).  This appears to be a nice contribution, and the idea may potentially generate future work.The two comments I might make are:1. It would be nice to have an example other than linear separators,to demonstrate that this general technique has interesting implications beyond what is already known.2. I suggest being a little careful about the claims in the paper.What is shown is that this upper bound on the sample complexity of this method is sometimes superior to the upper bound knownfor disagreement-based methods in terms of the disagreementcoefficient.  This is different from showing that the method sometimes achieves a sample complexity superior to that achievedby disagreement-based methods.


Provable Non-convex Robust PCA
We propose a new method for robust PCA -the task of recovering a low-rank matrix from sparse corruptions that are of unknown value and support. Our method involves alternating between projecting appropriate residuals onto the set of lowrank matrices, and the set of sparse matrices; each projection is non-convex but easy to compute. In spite of this non-convexity, we establish exact recovery of the low-rank matrix, under the same conditions that are required by existing methods (which are based on convex optimization). For an m×n input matrix (m ≤ n), our method has a running time of O r 2 mn per iteration, and needs O (log(1/ )) iterations to reach an accuracy of . This is close to the running times of simple PCA via the power method, which requires O (rmn) per iteration, and O (log(1/ )) iterations. In contrast, the existing methods for robust PCA, which are based on convex optimization, have O m 2 n complexity per iteration, and take O (1/ ) iterations, i.e., exponentially more iterations for the same accuracy.
Experiments on both synthetic and real data establishes the improved speed and accuracy of our method over existing convex implementations.

This paper is about developing non-convex counter part to Robust PCA. Robust PCA is a popular convex optimization scheme for decomposing a low rank matrix which is corrupted with sparse perturbations into its sparse and low rank components. However, the current methods based on convex optimization are computationally prohibitive as they require either matrix inversion and or full singular value decomposition. This paper develops a non-convex approach to this problem. The authors' simulations indicate that this non-convex algorithm can be much faster. The authors also provide some theoretical analysis for their algorithm.I think the subject of this paper is very timely and well suited for NIPS. Overall I find the paper to be interesting and well written. The technical issues I had raised have been resolved (for the most part) in the authors' response. There are still some issues with authors' claim of matching convex schemes (which I've explained below). However, I trust the authors to address this issue in the final version of the paper. Regardless of this issue I think this paper is a must accept at NIPS and I have updated my score. There also a couple of minor mistakes in the authors' response which I detail below.- Modified Pros:-- well written-- suited for NIPS-- new algorithm-- study of non-convex problems (very few such studies in the literature)-- real data experiments- Modified Cons:-- proven in an easier regime compared to convex counter parts-- need knowledge of problem parameters such as rank and incoherenceDetailed comments regarding the authors' response:1) ``We match KHZ 11"The authors do not match the deterministic results of HKZ11. They match it for the case where one is interested in sparsity per row/column. The interesting part about the deterministic result of HKZ is that their conditions can not only give this result but also nearly match the results of CLMW11 for the random models.2) ``Comparison with CLMW11:"Even with these added assumptions the author can not at this point match the results of CLMW11. My point is that I think the authors should be careful in their comparison and claims. That being said, I completely agree with the authors that these results can be matched with significantly more evolved proofs.


Analysis of Brain States from Multi-Region LFP Time-Series
The local field potential (LFP) is a source of information about the broad patterns of brain activity, and the frequencies present in these time-series measurements are often highly correlated between regions. It is believed that these regions may jointly constitute a "brain state," relating to cognition and behavior. An infinite hidden Markov model (iHMM) is proposed to model the evolution of brain states, based on electrophysiological LFP data measured at multiple brain regions. A brain state influences the spectral content of each region in the measured LFP. A new state-dependent tensor factorization is employed across brain regions, and the spectral properties of the LFPs are characterized in terms of Gaussian processes (GPs). The LFPs are modeled as a mixture of GPs, with state-and regiondependent mixture weights, and with the spectral content of the data encoded in GP spectral mixture covariance kernels. The model is able to estimate the number of brain states and the number of mixture components in the mixture of GPs. A new variational Bayesian split-merge algorithm is employed for inference. The model infers state changes as a function of external covariates in two novel electrophysiological datasets, using LFP data recorded simultaneously from multiple brain regions in mice; the results are validated and interpreted by subject-matter experts.

SUMMARYThis paper reports a bespoke variational scheme for inverting state-space models of spectral density implicit in LFP time-series.  In particular, its contribution is to provide a bespoke variational inversion using a hidden Markov model of discrete brain states that generate activity, where the form of spectral responses provides a Gaussian process model for the time-series.  I thought that this was an interesting if colloquial application of variational Bayes that may need to be contextualized within the broader church of dynamical causal modelling – and its application to neurobiological time-series.COMMENTS TO AUTHORSI enjoyed reading this interesting and detailed description of a variational state-space model inversion for time-series.  Technically, this was an impressive piece of work.  My main suggestions would be to contextualize this within the broader church of dynamic causal modelling and highlight the potential usefulness of your scheme.  Perhaps you could consider the following:1)	In the neurosciences, the variational inversion of state-space models of electrophysiological time-series is usually described in terms of dynamic causal modelling.  In fact, there is a literature on the dynamic causal modelling of cross spectral density that has been applied to multi region LFP time-series (and MEG).  I think it would be scholarly to look at this work.  You can find an overview of dynamic causal modelling at:http://www.scholarpedia.org/article/Dynamic_causal_modelingYour special contribution is a state-space model that is formulated in terms of a hidden Markov model.  This contrasts with usual DCMs that are based upon differential equations. You might want to highlight this because it is particularly useful for things like steep staging or endogenous transitions among different brain states.  2)	Your description of the generative model is framed for a machine learning audience (with things like spectral mixture kernel and Gaussian processes).  However, your rhetoric may confuse people in engineering and signal processing (and neurobiology).  It would be useful to link your terminology to more standard concepts (perhaps in a glossary).  For example, the Fourier transform of your spectral mixture is simply the auto-covariance function.  Furthermore, your use of the word kernel is colloquial.  In other fields, the kernel will be taken to mean the impulse response function or first order Volterra kernel whose Fourier transform is the transfer function (that corresponds to the spectral mixture).  It might be useful to clarify terminology here?3)	When you introduce the bound on model evidence or marginal likelihood, could you describe this as the variational free energy?  This will enable people to see the connections between the use of variational free energy in dynamic causal modelling and in your application.4)	To illustrate the potential usefulness of your approach perhaps you could make more of the clusters implicit in the sleep data.  Perhaps with something like:“To illustrate the potential importance of our (Bayes-optimal) states-space model inversion, we can now relate the clusters identified during sleep to classical sleep staging schemes (four distinct states).  By examining the similarity between the clusters (spectral mixtures) we identified and the classic spectral profiles, we can see how stage four can be decomposed into three sub-stages……”I am not sure how you would do this but it would be very nice if you could provide a proof of principle that your approach can take us beyond what we already know.MINOR POINTS1)	In the abstract, I would say: “The model is able to estimate the number of brain states….”2)	On page 2, it is not clear which of “the above two methods” you are referring to.  Can I suggest you say:“More recently new methods for tensor factorisation have been developed: in reference 7, tensor factorisation was applied to short-term FFT………….”This will make it clear that the tensor factorisation does not refer to the current paper.  3)	Below Equation 6, I would say the parameters describe the auto-correlation content associated with each y”.  I know what you mean but you are actually characterising data in the time domain not the spectral domain.  In other words, you are using spectral mixtures to provide constraints on the Gaussian process.4)	At the top of page 8, it was not clear to me exactly what was being predicted by the results of Table 1.  When you talk about a held out log predictive probability for different priors.  What was this probability distribution over?5)	Finally, I think you need to address a crucial issue in your generative model.  Usually, state-space models of spectral density (or auto covariance functions) accommodate cross spectra or cross covariance functions.  In other words, it is not just the spectral density at each node or region but the coupling between regions that is predicted on the basis of connectivity among regions.  I think you need to make it clear that your generative model does not consider complex cross spectra (cross covariance functions) and that – in principle – you could extend the generative model in this direction.6)	In the supplementary material, when talking about the updates for global probability vectors, you might want to mention that the use of point estimates means you do not have to consider the entropy of the posterior distribution implicit in the variational free energy (and that you can use the log posterior directly).  I hope these comments help should any revision be required.


Distributed Bayesian Posterior Sampling via Moment Sharing
We propose a distributed Markov chain Monte Carlo (MCMC) inference algorithm for large scale Bayesian posterior simulation. We assume that the dataset is partitioned and stored across nodes of a cluster. Our procedure involves an independent MCMC posterior sampler at each node based on its local partition of the data. Moment statistics of the local posteriors are collected from each sampler and propagated across the cluster using expectation propagation message passing with low communication costs. The moment sharing scheme improves posterior estimation quality by enforcing agreement among the samplers. We demonstrate the speed and inference quality of our method with empirical studies on Bayesian logistic regression and sparse linear regression with a spike-and-slab prior.

Summary:  The authors propose a new Bayesian inference algorithm that operates on subsets of data in a distributed environment.  This work is in the spirit of an active area of recent work on parallel MCMC for Bayesian inference using subsets of data.  The authors focus on comparisons to parallel methods; a complementary line of work uses data subsets as mini-batches. (I think these are not discussed, though they could be.) The new method, eventually called Distributed Context-Aware Sampling (DCAS), combines MCMC and expectation propagation.  All these methods must eventually share information between parallel cores, each of which "owns" a data subset.  The strategy here is to share some information between nodes, encoded in a "context prior."  In the experimental evaluation, the proposed method looks better than some other recent methods, but the experiments are very limited.      Quality:  The proposed method is interesting and well-motivated.  Theoretical results would have been nice, but not necessarily required -- in which case I would expect thorough and impressive experiments.  The algorithm relies on Normal models, fitted at each node, to share information between nodes.  A major question I have is, in the case where everything is actually Gaussian, does the algorithm converge to the exact target?      The writing is mostly good, except in the Intro, which appears to have been slapped together -- after a short but clear Abstract, the Intro begins immediately with a series of grammatically incoherent sentences containing embarrassing errors.  The citations, especially in the first paragraph, are dropped in and lack context (ironically).  Please explain the differences between these citations, and point out which concern Bayesian posterior sampling and which do not.      The Experiments are designed thoughtfully, and effort has been made to present a fair comparison of different sampling procedures, where it isn't obvious how to compare them directly.  The synthetic problem is nice (Bayesian logistic regression, 4,000 and 10,000 data points, 20 dimensions), but the evaluation could have benefitted from studying several problems along several axes -- synthetic and real data, larger datasets (which are free for synthetic models).  I would have liked to have seen standard MCMC convergence tests.         Clarity:  Since you are introducing the notion of the context prior, please italicize it in the Abstract.  It would also be useful to give some intuition there for what it means and why it is useful. Also in your Abstract, you could elaborate on your results. "We demonstrate the advantages of our algorithm" -- With respect to what?  Measured how?  Using how many cores?  What would be a summary of your quantitative results?  Your Abstract and Intro should introduce the name of your algorithm (DCAS), especially since it is not in your title.      Please define acronyms such as KDE and RKHS in the Intro.      The context prior is a nice idea and I appreciated its motivation with respect to the SVB paper.  In lines 068-069 where you say, "We believe ... it appears intuitive to suggest ... " -- Can you make stronger statements?  Or just explain that you are taking a reasonable Bayesian approach.        There is a lot of notation -- p's and q's with various superscripts and subscripts.  I wonder if it can be made clearer or more intuitive.  Perhaps there could be a little table, even as an appendix, summarizing all notation.  The superscripts (I think they are chi and iota) are not terribly intuitive -- or is there an explanation?  chi because it looks like an "x" for conteXt?  It took me a while to internalize that the subscript notation refers to a data subset and is also used to suppress this explicit dependence in the usual probabilistic notation -- e.g., p_i(theta) replaces p(theta | D_i).  I found this confusing on my initial reading because this led me to interpret the notation for subset posteriors as priors.        Footnote 2 appears at the end of a mathematical expression and looks like an exponent.      Thank you for the sentence around line 087, "To recap ..."      Please define "proj" in Eq. (7).      Section 2.4 could use some high-level sentences at the beginning, explaining the layout of the section.        In 2.4.1, line 132, I think you are defining p_i(theta) here, so please make this clear, as I was searching for it in the previous notation.  Also, calling this the "target distribution" may be confusing, since it isn't the true target.      Question about Algorithm 1:  should the 1/(T_i - 1) terms be 1/T_i?      Line 161, "As the underlying message passing progresses" -- this is confusing, as you haven't yet described the message passing component of your algorithm, which comes in the next section, 2.4.2.  This would be addressed by giving a high-level overview at the beginning of Section 2.4.        Some of the text in Figure 1 is way too small.      The descriptions of the synchronous, asynchronous and decentralized algorithms are very concise but could be expanded somewhat for clarity.  What explicit parallel architecture do you have in mind?  Who is receiving the messages?  Is there a master node?  It seemed that the implementation uses shared memory, available to all nodes, to share/update global information.  The description of the asynchronous scheme should make explicit how the latest historical copy of q_i^iota is used, using the notation introduced in line 185.      2.4.4, line 226 -- Where did tilde{f} come from?        Line 230, "Several possible remedies have been proposed" -- by whom?  You?      Experiments section -- please give explicit details about the cluster architecture -- you mention 8 nodes, is it 1 core per node?  It sounds like there is shared memory?  I assume then there is no need for MPI?  There are 8 nodes, but you study m = 8, 16, 24, 36, 48 partitions -- this suggests to me you have multiple cores per node.  The compute architecture needs to be clarified.      Make explicit that you do not evaluate the decentralized scheme.        It would be useful to explicitly describe the combining rules you use for SCOT, NEIS and WANG; this could go in Supplemental.  For WANG, you use their sequential sampler -- the corresponding paper proposes at least 3 sampling procedures, why did you choose this one?  What might you have expected to be different if you had used one of their other procedures?        Line 261, "Upon convergence" -- according to what criteria?      Please label the axes of Figures 3 and 5 (# samples).       Please make explicit how the ground truth samples are used, and precisely what KL-divergence is measuring (presumably between distributions fit to the ground truth samples and to the various MCMC samples.      If you feel you would benefit from more room, I would suggest moving (parts of) section 3.4 to Supplemental.       Originality:  To the best of my knowledge, the proposed method is novel.  This work has the same motivations as recent parallel, approximate MCMC schemes that operate on subsets (partitions) of data.  These other schemes either sample in an embarrassingly parallel manner, and then need to "fix up" the samples by combining them somehow, or have each parallel core sample from an approximate target (Weierstrass sampling). In all these methods, information must eventually be shared between nodes.  This work uses the novel idea of the context prior, which is continuously updated during execution.       Significance:  There is much recent interest in the area of scaling MCMC sampling by subsampling (partitioning) the data, and having separate parallel cores process each subsample.  Therefore the proposed algorithm is of interest and timely.  However, the paper presents no theoretical analysis or guarantees and therefore relies on empirical studies.  The empirical evaluation presented here compares with several other recent techniques, but is limited.  I really wanted to see a more enlightened discussion of this method compared to other (recent) work -- several other of these methods rely on Normal models, and a discussion/comparison would be very useful.       Other:  At least 4 of your references cite arxiv versions when (sometimes recent) journal or conference versions are now available [2, 4, 8, 15].      Line 067, "prompts our core" (delete "us to")       Line 081, commas around "e.g."      Typo around line 090, ", And ..."      Line 095 "two" not "2"      Line 103, "progress"      Line 104, "rules (4) naturally reduce"      Line 110, "first" not "firstly"      Line 215, explicitly define your notation for the zero matrix      Line 226, "not much of"      Line 266, "draw" not "draws"


Exponential Concentration of a Density Functional Estimator
We analyze a plug-in estimator for a large class of integral functionals of one or more continuous probability densities. This class includes important families of entropy, divergence, mutual information, and their conditional versions. For densities on the d-dimensional unit cube [0,1] d that lie in a β-Hölder smoothness class, we prove our estimator converges at the rate O n − β β+d . Furthermore, we prove the estimator is exponentially concentrated about its mean, whereas most previous related results have proven only expected error bounds on estimators.
α → 1 limit case), and L p divergence. Divergence estimators can be used to extend machine learning algorithms for regression, classification, and clustering from the standard setting where inputs are finite-dimensional feature vectors to settings where inputs are sets or distributions [22,18]. Entropy and mutual information (MI) can be estimated as special cases of divergences. Entropy estimators are used in goodness-of-fit testing [5], parameter estimation in semi-parametric models [33], and texture classification [6], and MI estimators are used in feature selection [20], clustering [1], optimal experimental design [13], and boosting and facial expression recognition [25]. Both entropy and mutual information estimators are used in independent component and subspace analysis [10,29] and image registration [6]. Further applications of divergence estimation are in [11].
Despite the practical utility of density functional estimators, little is known about their statistical performance, especially for functionals of more than one density. In particular, few density functional estimators have known convergence rates, and, to the best of our knowledge, no finite sample exponential concentration bounds have been derived for general density functional estimators. One consequence of this exponential bound is that, using a union bound, we can guarantee accuracy of multiple estimates simultaneously. For example, [14] shows how this can be applied to optimally analyze forest density estimation algorithms. Because the CMI of variables X and Y given a third variable Z is zero if and only X and Y are conditionally independent given Z, by estimating CMI with a confidence interval, we can test for conditional independence with bounded type I error probabilty.
Our main contribution is to derive convergence rates and an exponential concentration inequality for a particular, consistent, nonparametric estimator for large class of density functionals, including conditional density functionals. We also apply our concentration inequality to the important case of Rényi-α CMI.

QUALITYThe results are interesting, relevant for practice (e.g. for designing conditional mutual independence tests) and the technical development is o.k. as far as I could check. I do have two issues though (one which is related to the 'as far as I could check' - there are various unclarities in the math part of the paper, see below).(1) you claim that your results can be applied to Renyi and hence KL divergences, but these are *not* of the form (3) because p_1 and p_2 are defined on the same set X_1 there , so taking a product integral makes no sense. I suspect that this can be handled using similar techniques as you use in Section 6 for CMI estimation, which is also of different form; but you should definitely point out in the beginning of the paper that some modifications are needed; this had me confused for quite a while. (2) there is a step in the argument in Section 6 (which is crucial given the paper's motivation) which I really can't follow, on the bottom of page 7: first, where does the |1 - \alpha| in the formula come from/go to? Second, it seems you use the triangle inequality and the mean value theorem in a single step. This needs to be separated (and probably decomposed in smaller steps even further). Where does the \kappa_2 and \log \kappa* come from? I really don't get it. In fact my recommendation below is conditional on getting more information about this derivation in the author feedback. CLARITYThe goals and presentation are clear. I do have an issue with the notation of the main formula, i.e. the display on page 1 and eq. (3): why do you write p_1(x), ..., p_k(x) there and not p_1(x_1), ..., p_k(x_k) and then  explain that the x in dx stands for x = (x1, ..., x_k). The current definition makes, strictly speaking, no sense (p_i is a function of x_i, not x !). Am I correct or did I miss something? (Again my positive recommendation below is conditional on getting a reply to this question!)ORIGINALITYModerate - this is a step further in a direction of which it has recently become apparent that it is both significant and that interesting results are obtainable. SIGNIFICANCESubstantial - it is definitely useful to have estimates with exponential concentration.FURTHER REMARKS- Why is eq. (11) needed? Isn't this just a repetition of (9)? Better use this space for more explanation about what goes on further below...- When you refer to Renyi divergences: perhaps good to quote Van Erven and Harremoes' recent IEEE paper, which contains 'all you ever wanted to know about Renyi divergence' and seems to be rapidly becoming a standard reference (note I am neither Van Erven nor Harremoes)- I wonder whether the following papers are relevant, you might want to check them out. They deal with countably infinite outcome spaces but otherwise similar concerns as yours, and predate most of what you cite (again, I am not one of the authors :-). On the lower limits of entropy estimation.AJ Wyner, D Fostersubmitted to IEEE Transactions on Information Theory (apparently never officially published!?) Nonparametric entropy estimation for stationary processes and random fields, with applications to English textI Kontoyiannis, PH Algoet, YM Suhov, AJ WynerInformation Theory, IEEE Transactions on 44 (3), 1319-1327- The final sentence of the second paragraph of page 2 is grammatically incorrect and incomprehensible UPDATEAfter having seen the author's response, I updated my score to a 7. I would still like to insist though that my issue with Eq. (3) (see under CLARITY above) is explicitly dealt with in the final paper; either correct the formula (if I am correct) or explain in more detail how to read the formula (if I was wrong and the formula was correct) (the authors remained silent about this in their reply).


Robust Bayesian Max-Margin Clustering
We present max-margin Bayesian clustering (BMC), a general and robust framework that incorporates the max-margin criterion into Bayesian clustering models, as well as two concrete models of BMC to demonstrate its flexibility and effectiveness in dealing with different clustering tasks. The Dirichlet process max-margin Gaussian mixture is a nonparametric Bayesian clustering model that relaxes the underlying Gaussian assumption of Dirichlet process Gaussian mixtures by incorporating max-margin posterior constraints, and is able to infer the number of clusters from data. We further extend the ideas to present max-margin clustering topic model, which can learn the latent topic representation of each document while at the same time cluster documents in the max-margin fashion. Extensive experiments are performed on a number of real datasets, and the results indicate superior clustering performance of our methods compared to related baselines.

The paper introduces max-margin Bayesian clustering (BMC) that extends Bayesian clustering techniques to include the max-margin criterion. This includes, for example, the Dirichlet process max-margin Gaussian mixture that relaxes the underlying Gaussian assumption of Dirichlet process Gaussian mixtures by incorporating max-margin posterior constraints, and is able to infer the number of clusters from data. The resulting techniques (DPMMGM and a further one classed MMCTM) are compared to a variety of other techniques in several numerical experiments.The paper combines two clustering approaches: Deterministic and Bayesian clustering. Deterministic approaches can easily handle constraints, but lack the ability to infer the number of clusters. Bayesian techniques can infer the number of clusters, but not handle constraints such as the max-margin. Quality:The resulting BMC algorithm is claimed to be the first extension of regularized Bayes inference to the unsupervised clustering task. Furthermore, the DPMMGM algorithm uses the max-margin constraints to relax the Gaussian assumptions of the standard DPGMM, which is a strong new capability.Clarity:(pg 4, 184) “The max-margin constraints take effects in the model via \tilde\phi_i’s in (8).” Perhaps this claim was better illustrated in earlier work, but why this statement is true is not made sufficiently clear in this paper. The authors need to provide better support for these claims.(pg 4, 208) “Note that DPMMGM does not need the complicated class balance constraints [6] because the Gausses in the pseudo likelihood would balance the clusters to some extent.” Not sure that “Gausses” is an appropriate wording. This statement ending of “to some extent” is not sufficiently precise for this type of publication.(pg 6, 298) controll is misspelled(pg6, 284) Fig 2 shows that the two algorithms generate different results, but the discussion provided here does not give sufficient indication of why the result of the DPMMGM is better. DPGMM is claimed to be more fragmented, but why is that significant here, and, if anything, the DPGMM appears to be a better fit to the data, so what has been lost in switching to the DPMMGM?(pg6, 295) “thus driving the data points to collapse as well.” Pls clarify this statement. Please also expand this discussion to connect the intuition to the figure 3 (a)-(j). There are a lot of figures, a lot of trends, and each figure is difficult to interpret with the lines and clusters. The conclusion “the results indeed follow our intuition,” is far less obvious than the authors claim.Originality:While it would appear that the ideas are original, much of section 3 on robust Bayesian max-margin clustering follows [17] very closely. In particular the key steps between equations (6)-(7)-(8) reuse the techniques introduced in [17]. This suggests the possibility that the core ideas are in the prior work, and the authors are just re-applying them here to a new model formulation, which calls into question the novelty and importance of this work. Without further clarification, this influences the impact score. Significance:The extensive numerical results show that DPMMGM outperforms DPGMM, and similarly with MMCTM vs. SVM and S3VM, which appear to be significant results. However, while the paper contains a lot of material on model development, there appear to be very few theoretical statements on the performance or convergence of the algorithms.


Fundamental Limits of Online and Distributed Algorithms for Statistical Learning and Estimation
Many machine learning approaches are characterized by information constraints on how they interact with the training data. These include memory and sequential access constraints (e.g. fast first-order methods to solve stochastic optimization problems); communication constraints (e.g. distributed learning); partial access to the underlying data (e.g. missing features and multi-armed bandits) and more. However, currently we have little understanding how such information constraints fundamentally affect our performance, independent of the learning problem semantics. For example, are there learning problems where any algorithm which has small memory footprint (or can use any bounded number of bits from each example, or has certain communication constraints) will perform worse than what is possible without such constraints? In this paper, we describe how a single set of results implies positive answers to the above, for several different settings.

The paper considers the effect of memory constraints on some classes of online and distributed algorithms. The main goal of the authors is to show that there exist learning problems where imposing a memory constraint provably hurts the performance of any algorithm, in the sense that the best achievable performance guarantees are worse than some known upper bound on an algorithm that operates without memory constraints. The authors also provide a lower bound on the regret of online algorithms that operate under a specific feedback constraint. The results of the paper follow from an elegant information-theoretic argument concerning "hide-and-seek problems" where a learner has to detect a biased coordinate of i.i.d. random vectors by sequentially sampling a bounded number of coordinates.The paper is written very well and the technical quality is excellent. To my best knowledge, the main results of the paper (Theorems 2 and 3) are novel, and so are their implications in the considered learning settings. My main problem is that these settings are not general enough to draw significant conclusions as done by the authors. Generally, I find that the presented results are a bit oversold, and the paper would benefit from a more modest presentation. I list my main observations considering the contributions below.First, the lower bounds on the regret achievable by memory-bounded online learning algorithms seem to directly follow from the results of Seldin, Bartlett, Crammer and Abbasi-Yadkori (ICML 2014), whose counterexample is identical to the one presented in this paper. While this alone would not be a problem, it seems that the bound presented in Theorem 4 is only meaningful when the adversary is restricted to play binary losses (or at least choose from a finite pool of loss vectors). Indeed, allowing the learner to observe \Omega(d\sqrt{T}) bits of the loss of a *fixed* arm in each round is clearly not enough for any algorithm to achieve consistency, yet the lower bound of Theorem 4 fails to capture this intuition as it only guarantees a regret of \Omega(1). Theorem 4 is also clearly loose in finite-outcome partial monitoring games with minimax regret of \Theta(T^{2/3}). That is, the claim that Theorem 4 "quantifies in a very direct way the cost of information constraints for such learning problems" (line 303) is quite misleading.The examples considered in Sections 4.2 and 4.3 theoretically support the intuition that some learning problems are difficult to solve if the learner is forced to work with a memory budget that is significantly smaller than the natural representation of the problem instance to learn. Although it is nice to see such a result proven formally, this is hardly surprising: The same kind of information vs. performance trade-off can be observed when comparing the minimax performance guarantees concerning multi-armed-bandit and online prediction problems. While this contribution is somewhat stronger than the results of Section 4.1, I don't feel that the lesson is as fundamental as the authors claim.Detailed comments-----------------159: "Let X^t be a batch of i.i.d. instances" -- instances of precisely what? 172,173: correspond -> corresponds311: It would be interesting to contrast these results to those of Van Erven, Kotlowski and Warmuth (COLT, 2014), who propose a learning algorithm based on dropping out elements of each loss vector with an arbitrary fixed probability. When setting the dropout probability sufficiently high, one can guarantee that at most b bits are retained in each round with high probability, yet their algorithm succeeds in obtaining regret guarantees of O(sqrt{T log d}). I wonder if this contradicts the results of Theorem 4...?384: 1 - exp(-\Omega(...)) -> \Omega(1-exp(-...))407: While any b-memory protocol is indeed a (b,k,[m/k]) protocol, the converse is not true, thus this reduction doesn't seem to work.678, 825: x_j is missing from the conditions of Pr_0(.|.)


Feedback Detection for Live Predictors
A predictor that is deployed in a live production system may perturb the features it uses to make predictions. Such a feedback loop can occur, for example, when a model that predicts a certain type of behavior ends up causing the behavior it predicts, thus creating a self-fulfilling prophecy. In this paper we analyze predictor feedback detection as a causal inference problem, and introduce a local randomization scheme that can be used to detect non-linear feedback in real-world problems. We conduct a pilot study for our proposed methodology using a predictive system currently deployed as a part of a search engine.

The authors study situations in which predictions influence future inputs ofthe predictor, thereby introducing feedback into the system. The authors assumethat the predictor is linear in terms of its inputs, and model the feedback asa term that is added to the new prediction at $t+1$ and may depend nonlinearlyon the old prediction at $t$. The authors show that under these assumptions, itis possible to estimate the feedback function by adding independent noise tothe prediction. The method is similar in spirit to the "randomization" used inrandomized clinical trials, or in other situations in which the goal is toidentify a causal effect (instead of merely estimating statisticalassociation). The independence of the added prediction noise allows one toestimate the desired causal effect (which is the feedback, in this case).Although it is always difficult with articles like these to judge the novelty(as similar ideas might have come up in very different disciplines, rangingfrom econometry, engineering, control theory to causality as a subdiscipline ofAI and/or statistics), I haven't seen similar ideas before in the context oflearning systems. The most relevant related work that addresses feedback loops isall the work about "position bias" in the learning-to-rank literature. A common way to address feedback loops is to use interleaved comparison methods. This exploits the factthat if the prediction is a ranking, one can compare two different rankers by only outputting a single ranking (which combines, in an interleaved way, the rankings obtained by the two rankers that are compared), thereby allowing to partially observe factual and counterfactuals simultaneously.Nevertheless, the problem setting is interesting and the way in which it is formalized seems to be original. The analysis by the authors seems technically correct (although I didn't check the Theorems in detail) and I enjoyed reading the article. As I think the problem of feedback in live predictor systems is an interesting one that may becomemore relevant in the future (as machine learning gets more widely deployed) andmay deserve more attention in the NIPS community, I recommend acceptance.Two aspects with respect to which the article can still be improved are:(1) convincing the reader more of the significance of the problem that is addressed;(2) being more explicit about the assumptions that are made. For example, it would be very instructive to show how one could think about the additive-feedback assumption as a firstorder approximation (by some Taylor expansion argument). Also, what assumptions are made about the "natural process" $x^{(t)}$? It seems to me that the authors assume some kind of Markov property. In lines 211-214, the authors make the assumption that the predictor is linear. Is that assumption really necessary, or is the only relevant starting point for their analysis the equation in line 212-213 (basically, the additive feedback equation), which could possibly be interpreted as a first-order approximation of some more complicated nonlinear system (thereby allowing for nonlinear predictors)?Some small comments:o Why call this "statistical feedback"? Wouldn't something like "predictorfeedback" be a more descriptive term?o Lines 44-45: "This paper proposes a fully automatic method for detectingstatistical feedback loops that does not require modeling the feedbackmechanism." This sentence is misleading, as the authors DO model the feedbackmechanism (as an additive term that may depend nonlinearly on the lastprediction, see line 211). Another assumption about the feedback process thatthe authors make is that the prediction $\hat y^t$ only directly influences$x^{t+1}$ (it could also directly influence $x^{t+i}$ for any $i > 1$). Itherefore suggest reformulating or weakening this claim.o Line 47: "If future model predictions increase when we add artificial noiseto the current ones, then our system has a feedback problem." This sentence isconfusing. It would help to replace the word "increase" by "change" (becausethe sign of the change depends on the noise value - even assuming a positivecorrelation between noise and feedback, then the future model predictions onlyincrease when the noise is positive, and they decrease when the noise isnegative). Also, it is not clear why the feedback is a "problem". As shown inthe pilot study, feedback is not necessarily a problem (although it may becomeone if the feedback is too strong).o Line 211: what do you mean by "feedback function in $x$-space"?o Line 213: shouldn't this be $f(\cdot) = w \cdot f_{(x)}(\cdot)$?o (10): for consistency, use $\phi_N$ instead of $\phi_{\sigma_\nu}$


Mondrian Forests: Efficient Online Random Forests
Ensembles of randomized decision trees, usually referred to as random forests, are widely used for classification and regression tasks in machine learning and statistics. Random forests achieve competitive predictive performance and are computationally efficient to train and test, making them excellent candidates for real-world prediction tasks. The most popular random forest variants (such as Breiman's random forest and extremely randomized trees) operate on batches of training data. Online methods are now in greater demand. Existing online random forests, however, require more training data than their batch counterpart to achieve comparable predictive performance. In this work, we use Mondrian processes (Roy and Teh, 2009) to construct ensembles of random decision trees we call Mondrian forests. Mondrian forests can be grown in an incremental/online fashion and remarkably, the distribution of online Mondrian forests is the same as that of batch Mondrian forests. Mondrian forests achieve competitive predictive performance comparable with existing online random forests and periodically retrained batch random forests, while being more than an order of magnitude faster, thus representing a better computation vs accuracy tradeoff.

In this paper, the authors introduce Mondrian forests, a classifier inspired by random forests, Mondrian processes, and interpolated Kneser-Ney (IKN). They define Mondrian trees (MT), show how a forest is built from such trees, discuss relations to Mondrian processes and hierarchical Pitman-Yor processes (HPYP). They describe how online training and prediction work. Finally, they show that Mondrian forests provide competitive predictive accuracy as a function of the amount of training data seen and some of the best predictive accuracy as a function of amount of time spent. They discuss briefly how this accuracy breaks down when too many useless features are input and how future work might ameliorate this issue.Overall the model in this paper is interesting, the experiments are convincing, and the application to streaming data in particular seems timely and important. I definitely would like to see this research published.My major qualm with this manuscript is that it reads as though the authors have tried to fit a journal paper into the space allotted by NIPS. As it is, the paper is too dense, and still too many statements are left undefined or unjustified. (It would make a great journal paper if all of these connections were filled in; I go over specific cases below.) That being said, I think this work would make an excellent conference submission if it were carefully pruned by focusing on only the core development needed to reach the experiments section. E.g. by: removing some of the excess tree description, simply defining the generative process for Mondrian trees (the authors might note the Mondrian process as inspiration without dwelling on the theoretical relations to Mondrian processes---at least not in the main text), defining and using IKN without all of the HPYP and NSP build-up (in particular since HPYP and NSP don't seem to be used in the experiments or for additional insight beyond the IKN connection), etc.Major points:* Line 064: Big O in probability is required here, rather than the existing deterministic Big O (cf. https://en.wikipedia.org/wiki/Big_O_in_probability_notation ). If the authors plan to keep this part as is, I recommend they say something brief about what O_p means as it is less standard in this community than Big O.* Line 078/079: That the distribution of the Mondrian tree is invariant to the order of data points is a statement that needs to be proved (with proof referenced here), at least in the supplementary material.* Line 081: The authors state the depth of the tree is "typically logarithmic in n." Elsewhere, they state complexity bounds for balanced trees. This depth statement needs to be justified if it remains in the manuscript. Ideally, the authors would demonstrate (with informative summaries and/or graphs and/or tables) how balanced the trees were that were found in the experimental runs. Again, this could happen in the supplementary material but should be referenced when it is used in the main text. If the trees are not uniformly balanced, it is worth providing an analysis for other cases as well, if possible.* Section 3: A long time is spent reviewing trees in general and rooted, binary trees in particular. This wouldn't be so bad except that short shrift is given to Mondrian processes and the definition of Mondrian trees. Readers in this audience are likely familiar with rooted, binary trees but are less likely to be familiar with Mondrian processes, which are not defined in this manuscript.* Line 159--163: Somewhere in this area it should be noted that one expects the model to be very sensitive to scaling (due to the role of the "linear dimension" here) and that the authors recommend scaling each dimension to [0,1]. Currently, [0,1]-scaling is just an offhand remark in the experiments section.* Line 169--174: Bayesian CART (Chipman, George, McCulloch 1998) should be mentioned here as well since variants of these ideas have already been used in that model.* Line 174--185: If mentioned, projectivity needs a proof, perhaps in the supplementary material. It's not immediately clear that the data dependence in MTs doesn't spoil projectivity but could be established within a few lines.* Line 203--207: If the authors keep the discussion of NSPs and PYPs, they should provide citations for these models when they are introduced. * Footnote 2: The stated relative tractability of NSPs needs a citation here.* Figure 2 caption: Half a page is too much text for a caption, which as a rule of thumb should not generally exceed one paragraph. The vast majority of this text should be in the main text---or at worst the supplementary material, despite the caption's appealing small font size.* Algorithm 4: It is worth noting somewhere in this area that the reason this extension works, at a high level, is the property of exponential distributions that: X ~ Exp(a), Y ~ Exp(b) implies min(X,Y) ~ Exp(a+b)* Section 6: Bayesian CART, and possibly related work such as BART, should appear at least briefly in the related work section.* Line 360: It would be good to see a citation for the stated popularity of this choice of M.* Line 361: Since \lambda = \infty in the experiments presumably implies that each data point is in its own leaf, this might be a chance to prune some discussion of \lambda from the manuscript and thereby shorten it. Alternatively, if the authors keep the discussion of \lambda, the authors should note the implications in the text of choosing \lambda = \infty.* Line 361: \gamma = 10 D: It would be helpful to use names for all parameters on this line instead of just symbols. Even better might be to provide a reference to where they were defined in the manuscript. \gamma is somewhat buried on line 210 as is.* Line 364: Why not run more random initializations and report error bars for each algorithm as well? If the error bars are so small as to not appear on the plot, the authors could mention this in the text or caption.Minor editing points:* Line 053: It seems like D_i should be (X_i, Y_i). The existing notation for D_{1:n} is clunky and doesn't degrade well for subsets of 1:n.* Line 106: "limit" -> "bounds"* Line 194: "encourages label" -> "encourages the label"* Line 194: "discuss the" -> "discuss how the"* Line 238: "rectngble" -> "rectangle"* Line 360: "As common" -> "As is common"


Tight Continuous Relaxation of the Balanced k-Cut Problem
Spectral Clustering as a relaxation of the normalized/ratio cut has become one of the standard graph-based clustering methods. Existing methods for the computation of multiple clusters, corresponding to a balanced k-cut of the graph, are either based on greedy techniques or heuristics which have weak connection to the original motivation of minimizing the normalized cut. In this paper we propose a new tight continuous relaxation for any balanced k-cut problem and show that a related recently proposed relaxation is in most cases loose leading to poor performance in practice. For the optimization of our tight continuous relaxation we propose a new algorithm for the difficult sum-of-ratios minimization problem which achieves monotonic descent. Extensive comparisons show that our method outperforms all existing approaches for ratio cut and other balanced k-cut criteria.

The paper presents a novel algorithm to solve the balanced k-cut problem. The k-cut criterion is defined by using the Lovasz extension of a set function. A relaxation of balanced k-cut functionals can be formulated using Lovasz extensions of cuts and of normalizing constraints. The appealing feature of this relaxation is the connection between convex Lovasz extensions and submodular set functions. The authors propose an algorithm that minimizes a sum of auxiliary variables that are lower-bounded by ratios. The minimization is further constraint by simplex and unique membership constraints. Clarity: The paper is very well written and it appropriately refers to prior art. Originality: There exists significant prior art and the novel idea modifies the optimization strategy. I consider the originality and in particular the novelty as limited but its significance due to the success in the experiments might justify publication of this paper at NIPS. Significance: Due to the surprisingly clear superiority of the new method, this paper might show high impact in the future despite its high similarity with prior art. Minor comments:Figure 1 has five subfigures but only four are explained in the caption. What is the fifth subfigure about?


Randomized Experimental Design for Causal Graph Discovery
We examine the number of controlled experiments required to discover a causal graph. Hauser and Buhlmann [1] showed that the number of experiments required is logarithmic in the cardinality of maximum undirected clique in the essential graph. Their lower bounds, however, assume that the experiment designer cannot use randomization in selecting the experiments. We show that significant improvements are possible with the aid of randomization -in an adversarial (worst-case) setting, the designer can then recover the causal graph using at most O(log log n) experiments in expectation. This bound cannot be improved; we show it is tight for some causal graphs. We then show that in a non-adversarial (average-case) setting, even larger improvements are possible: if the causal graph is chosen uniformly at random under a Erdös-Rényi model then the expected number of experiments to discover the causal graph is constant. Finally, we present computer simulations to complement our theoretic results. Our work exploits a structural characterization of essential graphs by Andersson et al. [2]. Their characterization is based upon a set of orientation forcing operations. Our results show a distinction between which forcing operations are most important in worst-case and average-case settings.

UPDATE AFTER DISCUSSION ROUND WITH AUTHORS:thanks for accepting discussing your paper with us. Really liked reviewing it !Still unclear to me when you state "Specifically, this result show that having many large cliques is not an insurmountable barrier." I would have thought that it is ?! See K. S. Sesh Kumar and F. Bach. Convex Relaxations for Learning Bounded Treewidth Decomposable Graphs. Technical report, HAL 00763921, 2012. Proceedings of the International Conference on Machine Learning (ICML) on the subject. A discussion in relationship to tree-width (in a future version of your paper) would really be a plus ! My understanding is that ONE dense and large clique is already a big problem...But in my framework, it's about structure network recovery, not to have an orientation from a graph skeleton...Best wishes !%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%This paper proposes to give conditions on the needed number of experiments to reconstruct a '"causal" graph from (interventional) data (given the skeleton is known from observational data). It follows from work of others ([1]). It states that only $ \log \log n $ experiments ($ n $ being the numbe of variables or nodes) are needed to recover the full causal structure in a "very adverse" situation (optimality of the result claimed) and this could fall down to a constant sample size when the graph is thought to be Erdös (is it really useful ??).Section 1 is an introduction, Section 2 gives background on graph, causality and the potential to reconstruct from observations and single perturbations. Section 3 introduces multiple random experiments so as to improve the actually needed number of experiments to be ale to decipher the causal structure. Section 4 gives a stringer result in a more constrained situation where graphs are generated according to an Erdös-Renyi model. Section 5 presents some simulation resultsOriginal results are clearly presented and seem to be an interesting step forward in a very important field: deciphering true causal relationships in complex systems and not only co-occurrence relationships. The paper is ok to read but some slight improvements could be made in the writing? A careful reading and the authors could be trusted to achieve a better readability !Few comments: - is it ok to put references in the astract in NIPS policy ? And to use CR when writing it ? - l24: yes, I remember a presentation on Rau et al. BMC Syst. Biol. which also mentioned that there are much cleverer perturbation than single KO experiments ?! This makes sense and can be related to the graph structure at hand. - last sentence: rephrase, this is unclear. - l37: dependences or dependencies ? - in [article missing] seminal work. - l43: sentence could be turned better. - try to avoid footnote. And footnote #1 is not a very accurate statement ?! - Some work of Guyon and colleagues may be cited. - why is "our results" section 1.1 and no 1.2 follows ? - I assume the 'adversary' notion is clear for those familiar with the game framework of [2] but not for the entire NIPS readership ?! - l61 "pre-commit" ?? - how do you framework compare to that of Peters & Bühlmann, Biometrika 2014 ? - would you say that your result on Erdös-Renyi graph is surprising ? I would say NO, their structure is relatively poor as compared to simulated but slighlty richer classes of models. Not to speak about true graph which obviously fall in none of these simulated framework (Erdös-Renyi, scale-free, small world...). - Table 1: I really beg you to avoid such a table ot present your results, they are difficult to read. Not on slides please ??!! And give a caption which has a signification please ! - A wee conclusion/discussion might be a plus...