MCMC for Variationally Sparse Gaussian Processes
Gaussian process (GP) models form a core part of probabilistic machine learning. Considerable research effort has been made into attacking three issues with GP models: how to compute efficiently when the number of data is large; how to approximate the posterior when the likelihood is not Gaussian and how to estimate covariance function parameter posteriors. This paper simultaneously addresses these, using a variational approximation to the posterior which is sparse in support of the function but otherwise free-form. The result is a Hybrid Monte-Carlo sampling scheme which allows for a non-Gaussian approximation over the function values and covariance parameters simultaneously, with efficient computations based on inducing-point sparse GPs. Code to replicate each experiment in this paper is available at github.com/sparseMCMC.

0. SummaryThis paper presents a method for sampling from the (approximate) posterior in variational sparse Gaussian process models with factorizing likelihoods. The main claimed contribution is that allowing a "free-form"for the approximate posterior, instead of constraining it to belong to a pre-specifiedfamily of distributions. 1. Clarity This is paper is well written with the contributions clearly stated, describing also how the the paper differentiates from previous work. 2. Quality, originality and significance The paper addresses a significant problem in probabilistic inference, that of approximating posterior distributions in models with Gaussian process priors and factorizing likelihoods, allowing efficient sampling of inducing variables and covariance hyper-parameters. The paper is technically sound, buildingupon previous work on variational approaches to sparse GP models (reference 5 in the paper) with the main contribution appearing in Equations (5) and (6),rewriting the variational objective so that one can use MCMC (HMC) machinery to to draw samples from the joint approximate posterior over inducing variables and hyper-parameters. I note in passing that Equation (5) seems to have typo, as the expectation of the conditional likelihood over the conditional prior should be exponentiated (?).The very same idea of sampling from the approximate posterior within the inducing-point framework has been proposed in the past (see Equation 1.27 and corresponding text in [1] below) but I am unaware of its implementation. I am surprised the authors do not reference this work. Nevertheless, the contribution of the paper is still significant as it carries out the development of a practical framework that does not involve the commonly used assumption of optimizing over the parameters of a fixed distribution, while still maintaining the computational efficiency of sparse GP models. I do not see major concerns about this paper, although it somehow gives the readers the impression that the proposed method is sampling from the true posterior of a sparse GP model, which I do not think is the case giving the assumed form for the posterior approximation in the denominator of Equation (3). It would be useful if the authors discuss this.The paper presents results on a wide variety of experiments on small datasets and on MNIST, comparing to the full Gaussian approximation and to 'gold standard' MCMC. It is interesting to see that there is not a large practical difference between a free-form approximation and a Gaussian approximation but I think the proposed approach it is still useful on its own right. On the MNIST data, it would be useful to know how long it took to run the whole experiment.  3. Minor comments - Figures 1 and 4 show some weird numbers '000, 001, ...' - line 370: "As Girolami and S' -> 'As Girolami and Rogers'- line 401: "approaches to to classify" -> "approaches to classify"4. References[1] . M. K. Titsias, M. Rattray and N.D. Lawrence. Markov chain Monte Carlo algorithms for Gaussian processes. Chapter to appear in the book "Inference and Learning in Dynamic Models"


Character-level Convolutional Networks for Text Classification *
This article offers an empirical exploration on the use of character-level convolutional networks (ConvNets) for text classification. We constructed several largescale datasets to show that character-level convolutional networks could achieve state-of-the-art or competitive results. Comparisons are offered against traditional models such as bag of words, n-grams and their TFIDF variants, and deep learning models such as word-based ConvNets and recurrent neural networks.

The paper presents a character-level convolutional network architecture and applies it to eight text classification problems on large datasets that the authors construct. It also presents comparative results from several word-based deep NN models as well as bag-of-ngrams models. The character-level ConvNets outperform word-based models on four out of eight datasets, when word-based data augmentation is used.The clarity and quality of writing are ok but the presentation of the method and results could have been much more clear. There are numerous grammatical and spelling errors.The originality is not very high since character-level models have been used for other tasks. The representation of the input as a fixed length vector of length 1014 characters is unintuitive, as one would like to have a story for handling documents of all lengths. The use of data augmentation using a thesaurus is not consistent with the stated advantage of character-level models: if words and even a thesaurus is available, then word-level tokenization can be performed. The experimental results contain a large number of comparison models as well as several variants of the proposed architecture: large vs small hidden layers, use of thesaurus augmentation, and lower-case characters versus both lower and upper-case ones.For four of the eight problems character-level models with data augmentation performed best and outperformed word-based models. In some cases character-level models without data augmentation outperformed word-level models as well. The data settings where character-level models performed best were also ones with user-generated content which might contain misspellings and ones with the largest number of training instances.Since the hyper-parameters of the different models were not tuned and there is no comparison to published state-of-the-art results (or models with state-of-the-art results), it is not clear what the significance of these experiments could be.  *****Thanks for the detailed response. It will be good to add the justification for dataset choice to the paper.


Optimal Ridge Detection using Coverage Risk
We introduce the concept of coverage risk as an error measure for density ridge estimation. The coverage risk generalizes the mean integrated square error to set estimation. We propose two risk estimators for the coverage risk and we show that we can select tuning parameters by minimizing the estimated risk. We study the rate of convergence for coverage risk and prove consistency of the risk estimators. We apply our method to three simulated datasets and to cosmology data. In all the examples, the proposed method successfully recover the underlying density structure.

The paper focuses on density ridge estimation. The authors propose the use of coverage risk which is a generalization of the mean integrated expected error. The authors propose two estimators (based on either smoothed bootstrap or data splitting), prove their consistency, and study convergence rates. Particularly nice is that, unlike some of the cited works, the proposed method does not suffer from the problems related to the notorious estimation of the smoothing parameter.Experiments are done on simulated data and Cosmic Web data.   Quality: The paper is technically sound, and its overall quality is high. The problem attacked is interesting and of importance.Clarity: The paper is well written and clear.Originality: As far as I am aware of, the idea presented is new. Significance: I believe this work will have a nice impact.  A remark on the experiments: In both Figures 3 and 4, the authors show under-smoothing and over-smoothing results next to theirs. That's fine, but what is really missing here is a comparison with what ones gets when one uses simple cross validation for the optimal smoothing parameter for standard Kernel Density Estimation (e.g., using a Gaussian kernel) and then does the ridge extraction after the fact. Presumably, according to the authors' claims this would be inferior to their method which finds the optimal smoothing parameter wrt the task at hand (ridge detection) as opposed to the optimal parameter for general-purpose KDE. Such an experimental result, whose absence is disappointing, would have strengthened the authors' main claim about the utility of the method and would have also convinced me to give their paper a higher grade.  A remark regarding principal curves:The authors may want to also cite Hastie's work from the 80's.  ----------------------------- ----------------------------- Having read the rebuttal, my impression remained positive. I keep my score as it is. That said, I am sightly concerned about the wording used in the rebuttal regarding my remark on cross validation: while they committed themselves in the rebuttal to add the results obtained using ordinary CV, they avoided stating whether this comparison indeed makes their method look favorable. I hope this is just a poor choice of wording and nothing more. It will be very disappointing to find out that the results are actually the same.


Lifted Inference Rules with Constraints
Lifted inference rules exploit symmetries for fast reasoning in statistical relational models. Computational complexity of these rules is highly dependent on the choice of the constraint language they operate on and therefore coming up with the right kind of representation is critical to the success of lifted inference. In this paper, we propose a new constraint language, called setineq, which allows subset, equality and inequality constraints, to represent substitutions over the variables in the theory. Our constraint formulation is strictly more expressive than existing representations, yet easy to operate on. We reformulate the three main lifting rules: decomposer, generalized binomial and the recently proposed single occurrence for MAP inference, to work with our constraint representation. Experiments on benchmark MLNs for exact and sampling based inference demonstrate the effectiveness of our approach over several other existing techniques.

Summary:   Lifted inference enables tractable inference in large probabilistic models by exploiting symmetries and independencies over variables. This paper provides a comprehensive method to (a) specify constraints; (b) use them for lifted inference; and (c) produce constraints from evidence. The authors first introduce setineq a constraint language for specifying symmetries and independencies using set membership and (in)equality constraints, and supply basic operators for manipulating these constraints. Next, The authors develop algorithms to use these constraints and basic operators to produce lifting rules (Decomposer/Binomial). Finally, the authors present a greedy method for deriving constraints from evidence. The authors test a Java-based implementation of the setineq-based constraint specification against other lifted inference implementations and show superior scaling and equivalent output to unconstrained representations.         Quality:   This paper has a number of technical contributions in representation and implementation of lifted inference methods. Experimental evaluation uses standard benchmark datasets, seems to provide fair comparisons to related work, and shows impressive performance.        Clarity:   My main issues with this paper relate to clarity. The paper relies heavily on the supplemental material for proofs, algorithms, and the Single Occurrence rule, and including some of this supplemental material in the main paper may be helpful.Since the paper covers a great deal of ground, each contribution must be framed carefully. The formal specification of setineq and the join/partition operators are clearly presented and benefits from many examples to help the reader understand how constraints are specified. However, the remainder of the paper would be improved with restructuring, proofreading (see below), and additional examples. One potential improvement would be moving Section 4.4 to section 3, since is relatively independent of the lifted inference rules and strongly relates to producing a constraint tuple from evidence. Additionally, revising section 4.4 may be helpful - the section is very terse and it's difficult to distinguish the paper's contributions from the previous work. In the lifted inference rules, the description of the Binomial algorithm (4.3) is extremely difficult to read. Examples for Theorem 4.1 and 4.2 would lend the preceding (somewhat dense) exposition some intuitive appeal.         Originality:   To the best of my knowledge, this work is original and relevant related work is cited.        Significance:   My understanding is that this provides an incremental refinement to lifted inference, and provides a starting point for deeper investigation. The empirical results seem very strong, and I wouldn't be surprised to see following work building more advanced lifted inference methods on setineq.        Typos:   Bell number (210)   grammar (240)   set definition (253)   set V^y_l (254)   argumene (297)   nmw (304)   Professors and Students (365)


Local Causal Discovery of Direct Causes and Effects
We focus on the discovery and identification of direct causes and effects of a target variable in a causal network. State-of-the-art causal learning algorithms generally need to find the global causal structures in the form of complete partial directed acyclic graphs (CPDAG) in order to identify direct causes and effects of a target variable. While these algorithms are effective, it is often unnecessary and wasteful to find the global structures when we are only interested in the local structure of one target variable (such as class labels). We propose a new local causal discovery algorithm, called Causal Markov Blanket (CMB), to identify the direct causes and effects of a target variable based on Markov Blanket Discovery. CMB is designed to conduct causal discovery among multiple variables, but focuses only on finding causal relationships between a specific target variable and other variables. Under standard assumptions, we show both theoretically and experimentally that the proposed local causal discovery algorithm can obtain the comparable identification accuracy as global methods but significantly improve their efficiency, often by more than one order of magnitude.

The goal of the paper is to identify direct causes and effects of a given variable (target variable). The main contribution stands in that the authors achieve their goal without the need of first discovering the whole causal graph before restricting attention to the neighborhood of the target variable. Instead, they directly focus on discovering the local causal structure.Existing algorithms can discover the Markov blanket (MB) of a target variable and a subset of it that consists of the set of parents and children (PC) of the target variable. The goal of this paper is to identify which of the nodes in PC are parents and which children of the target.Significance & Originality:I consider the contribution of the paper significant. It improves on efficiency and scalability compared to previous methods which need to perform the intermediate step of global causal discovery to achieve the same goal.Clarity: I think that lack of clarity and flow in several parts is a drawback of the paper. I found some descriptions confusing. In general, I would suggest more explicit explanations, motivating examples and high level explanations before going into the technical parts.Specifically:- Last paragraph of Introduction: the authors refer to Local Causal Discovery. It took me some time to realize that they refer to existing work and not to theirs, since this term coincides with the title of the paper. I would suggest changing the title to something more specific like "local causal discovery of direct causes and effects". Moreover, I find the name of the proposed method "Causal Markov Blanket" also too general and maybe misleading. The authors could probably consider a common name for both title and method involving "direct causes and effects".- Section 3: Some high level explanations are necessary before going into the technical details. The authors could provide a short high level summary of all steps of the proposed method in the beginning of section 3.2.- I could be missing something in Algorithm 2: as far as I understand, ID_T should contain the causal identities of *just* the PC set of T. But in step 1 it is mentioned "..the causal identities of *all* nodes.." and in Algorithm 2 the size of ID_T is |V| where V is the entire variable space and no just the PC.- The output of Algorithm 2 is the ID_T. How is this converted to a CPDAG? For id_T 1 and 2 it is clear. For id_T =3 do you draw an indirect edge? Providing some visual examples of the output CPDAG of the Algorithm would be helpful.- Elaborating more on why failing to meet Lemma 3 leads to Markov equivalent structures, could be useful.- The authors could consider improving the flow in page 3.- Some parts are not very specific/explicit: e.g.: * Sec. 4, first par.: "causal identities for a target node": isn't it "causal identities of nodes in the target's PC set"? * I understand what is a CPDAG but it would be helpful to define what it is meant by the "CPDAG of a target variable". Is it a subset of the original CPDAG containing T and the parents and children of it? *In the proof of Theorem 3: "CMB will identify up to a graph's CPDAG": is this again the subset? * Footnote 3 is not clear * Sec. 3.1: what does it mean "some V-structure children and spouses" ? * Sec 3.1: "PC set of a MB" : isn't it "PC set of a target's MB"? * The terms "local causal discovery of the target" and "local causal nodes of the target" could be rephrasedQuality:I have not checked all the technical details but the paper seems to have a well-supported theoretical analysis followed by experimental results. The results show significant improvements in efficiency compared to global methods.   -Typos: * There are several typos in the references section. * Introduction: in term -> in terms *Sec. 3.1: "... no selection bias, every.." -> "...no selection bias, that every..." * Experiments: "along its standard deviation" -> "along with its...", "As shown in in Table"-> remove one "in"


Convergence Rates of Active Learning for Maximum Likelihood Estimation
An active learner is given a class of models, a large set of unlabeled examples, and the ability to interactively query labels of a subset of these examples; the goal of the learner is to learn a model in the class that fits the data well. Previous theoretical work has rigorously characterized label complexity of active learning, but most of this work has focused on the PAC or the agnostic PAC model. In this paper, we shift our attention to a more general setting -maximum likelihood estimation. Provided certain conditions hold on the model class, we provide a two-stage active learning algorithm for this problem. The conditions we require are fairly general, and cover the widely popular class of Generalized Linear Models, which in turn, include models for binary and multi-class classification, regression, and conditional random fields. We provide an upper bound on the label requirement of our algorithm, and a lower bound that matches it up to lower order terms. Our analysis shows that unlike binary classification in the realizable case, just a single extra round of interaction is sufficient to achieve near-optimal performance in maximum likelihood estimation. On the empirical side, the recent work in [12] and [13] (on active linear and logistic regression) shows the promise of this approach.

The paper addresses the problem of active learning by trying to find the MLE from a parametrized class of models. It shows that under certain conditions an expression of the Fisher information is a suitable surrogate objective function. Optimizing this surrogate objective helps find a \emph{good} sampling strategy. However, this expression entails the optimal parameter \theta^* whose approximation is the ultimate goal of the algorithm. To overcome this issue the paper takes a two step approach. In the first step a fraction of the budgeted queries is used to get a rough estimate of \theta^* which is subsequently used to optimize the surrogate objective function and obtain a good sampling distribution. The paper shows that using this two step approach one can achieve an asymptotically optimal error given a fixed budget on number of queries for certain classes of models which include GLMs. The quality and presentation of the paper is good. The result is significant, at least from a theoretical perspective, albeit being slightly disappointing since it shows that a two step batch sampling is asymptotically as good as one can get with a full fledged active learning on these classes of models. This aspect brings to mind the question of applicability of the result to practical settings and strongly begs some numerical demonstration and comparison which is unfortunately missing in the paper.comments: 1. diameter(\Theta) in Theorem 1 is not defined. 2. The big O notation has in itself an implicit inequality and there is no proper definition of f > O(g) that I am aware of. Hence the author(s) should clarify the condition in Theorem 1 which states that m > O(...). 3. It would be informative to see what are the practical implication of Assumption 1 vis-ae-vis the result of Theorem 1. For example, if the set U was randomly chosen from \mathcal{X} with some reasonable distribution what is the scaling of \sigma_min in part 3 of Assumption 1 and what would be the implication for m_1. At its current form the paper does not give any idea under what regime this is an applicable method. The lack of any numerical demonstration even in the supplementary section is not helpful. While the theoretical contribution is certainly valuable demonstration of the applicability of the result makes it many folds more interesting. Furthermore, if it is possible to demonstrate the method in practical scenarios this type of result strongly begs some comparison with other methods.


Large-Scale Bayesian Multi-Label Learning via Topic-Based Label Embeddings
We present a scalable Bayesian multi-label learning model based on learning lowdimensional label embeddings. Our model assumes that each label vector is generated as a weighted combination of a set of topics (each topic being a distribution over labels), where the combination weights (i.e., the embeddings) for each label vector are conditioned on the observed feature vector. This construction, coupled with a Bernoulli-Poisson link function for each label of the binary label vector, leads to a model with a computational cost that scales in the number of positive labels in the label matrix. This makes the model particularly appealing for real-world multi-label learning problems where the label matrix is usually very massive but highly sparse. Using a data-augmentation strategy leads to full local conjugacy in our model, facilitating simple and very efficient Gibbs sampling, as well as an Expectation Maximization algorithm for inference. Also, predicting the label vector at test time does not require doing an inference for the label embeddings and can be done in closed form. We report results on several benchmark data sets, comparing our model with various state-of-the art methods.

This paper is about a new Bayesian method for multi label learning. The goal is to classify accurately in settings where there are many potential labels but only a few of them apply to each data point. The basis of the new results is a new generative model for the label vector of each example. Specifically the label vector y_n of the n-th example is generated as  y_n = f(V(\sigma(Wx_n)),where Wx_n is a lower dimensional projection of the n-th instance x_n, followed by an element-wise sigmoid activation \sigma. The final operation f corresponds to drawing Poisson random variables with rates given by V(\sigma(Wx_n)) and thresholding these so-called latent counts by taking the minimum with 1. The inference problem boils down to (heuristically) estimating various parameters of the model using Gibbs sampling and EM. In doing so, the authors observe that the latent counts are sufficient statistics of the model parameters. Evaluation: I found the generative model interesting and natural as it combines low-rank approximations with non-linearities in an interesting way. It can also be interpreted as a 3 layer neural net. I'm not sure however that inference in this model is really feasible on larger problems. The authors evaluate the approach on four tiny data sets. The largest one has fewer than 20k samples and one has as few as 40 test samples. It's not clear to me that these benchmarks are representative of real problems. The authors have yet to demonstrate that this approach will be competitive with, say, WSABIE on even medium size data sets (doesn't fit in memory but on a single machine). In particular, I don't follow the narrative in this paper which repeatedly emphasizes the "large-scale" or "massive data" aspect when there isn't even a medium data benchmark.At any rate, the multi label problem is also interesting for small data and in this regime the approach appears to be competitive with or better than state of the art. Complementing stronger experimental results, it would also be interesting to have more theoretical support for the inference method. Currently the approach is heuristic. Small remark:- The g() function in (10) appears to be a typo. If it's supposed to be there, please define it.


Large-scale probabilistic predictors with and without guarantees of validity
This paper studies theoretically and empirically a method of turning machinelearning algorithms into probabilistic predictors that automatically enjoys a property of validity (perfect calibration) and is computationally efficient. The price to pay for perfect calibration is that these probabilistic predictors produce imprecise (in practice, almost precise for large data sets) probabilities. When these imprecise probabilities are merged into precise probabilities, the resulting predictors, while losing the theoretical property of perfect calibration, are consistently more accurate than the existing methods in empirical studies.

This paper proposes two techniques for post-processing a score-valued output of a binary classifier by converting it into an estimate of a class probability.The techniques are simple and efficient. It is surely publishable in principle, however, in my perspective the paper in its current form has far too many weak spots to be acceptable. They are listed in the following.Until the conclusion the distinction between contribution and background remains fuzzy.The content of section 2 seems trivial, in particular proposition 2. All it takes is sorting and binary search. However, this seems to be one of the "major" contributions.The hyperparameters of the learning machines were set by minimizing the training error?! E.g., a Gaussian kernel SVMs can easily achieve zero error. Then all we see is over-fitting. This is a clear no-go, it renders all experimental results meaningless. That alone is a reason for rejection.The improvement over the baseline methods is consistent, but the effect size is small. So are the new methods relevant, in particular since they take more space and time than Platt's method? I am missing a discussion of this point. For acceptance I want to be convinced that somebody out there cares.Minor comments:In contrast to what it stated in the paper, Platt's method is not invariant w.r.t. the (sigmoidal?) conversion of the scores to the unit interval. It does not become 100% clear whether the conversion is actually performed with Platt's method or not.At some point the paper mentions that most models output values in the unit interval, which could hence be interpreted as probabilities. This point is not followed upon in the experimental evaluation. Why not?In section 5.2 it is claimed that there is no simple ad-hoc regularization for isotonic regression. I don't see why Platt's regularization technique cannot be applied, which basically amounts to adding two virtual scores of plus and minus infinity.The first word of the title reads "Large-scale". Actually, nothing in this paper is large scale, and I don't see any relation to processing of large-scale data.The plots in figure 1 are far too small to be readable in a standard b/w printout.


Adaptive Low-Complexity Sequential Inference for Dirichlet Process Mixture Models
We develop a sequential low-complexity inference procedure for Dirichlet process mixtures of Gaussians for online clustering and parameter estimation when the number of clusters are unknown a-priori. We present an easily computable, closed form parametric expression for the conditional likelihood, in which hyperparameters are recursively updated as a function of the streaming data assuming conjugate priors. Motivated by large-sample asymptotics, we propose a novel adaptive low-complexity design for the Dirichlet process concentration parameter and show that the number of classes grow at most at a logarithmic rate. We further prove that in the large-sample limit, the conditional likelihood and data predictive distribution become asymptotically Gaussian. We demonstrate through experiments on synthetic and real data sets that our approach is superior to other online state-of-the-art methods.

- 'Theorem 1', is a well-know result in the field. It should be easy to find references from any introductory lecture notes in the field. There seems to be nothing original about the proof, and proper attribution should be made anyways.- SUGS updates parameters (step 2) but they are never used since the parameters are marginalized via conjugacy in step 1. This could be misleading.- Moreover, the following argument starting at line 221 is hand-wavy and should be formalized since the following depends on it.- It should be clearly pointed out that the mean of the posterior on alpha as a point estimates is an ad-hoc, non-Bayesian procedure.- The argument in Section 4 assumes a parametric model. This should be clearly acknowledged as a limitation of the argument. Moreover the assumption 1 in the theorem seems fairly strong. The following paragraph mentions that they hold numerically, but the results do not seem very conclusive (given the large peak towards the end of the run). One or way or the other, the language should be changed, as it seems to suggest that their numerical results formally establish that the condition holds, which can never be strictly the case for an a.s. statement.- Claims of 'stability' (in the title of section 4): what is established in this section is that the number of cluster will not grow too fast under strong assumptions. This is not the key issue: one could just put a hard limit of alpha K1 log(n) + K2 if really needed. What one would expect stability to mean is a bound on the closeness of the greedy posterior to the true posterior that do not blow up exponentially in the number of step size. It is unclear that ASUGS has this property even from the numerical example given the small datasets considered.- There are large portions of the work that reproduce known results in the field. For example, section 5 is about standard results on matrix t-distributions, and much of the supplement contains standard, introductory level results in conjugacy. Unless there is something new there, a citation to existing in-depth tutorials (e.g. http://www.cs.ubc.ca/~murphyk/Papers/bayesGauss.pdf) would be more useful.- The evaluation is clearly rushed and incomplete. First, since the motivation is computational, a key missing piece is some experiments that provides information on the trade-off between posterior approximation quality vs. compute cost. - At least one classical, non-sequential methods should be included for completeness, e.g. the split-merge method of R. Neal.- Since an important motivation of the paper is scalability, it would be good to use the full 60k in Section 6.2. - There are not much useful information or insight gained in Section 6.2. More convincing data analysis would strengthen the paper.- Code availability should be clarified. More info on the experimental setup should be added at the very minimum for reproducibility purpose.  Minor presentation issue: 'define the auxiliary variables': I would avoid calling these auxiliary variables, since auxiliary variables have a distinct, specialized meaning in computational Bayesian statistics.


Fast Randomized Kernel Ridge Regression with Statistical Guarantees *
One approach to improving the running time of kernel-based methods is to build a small sketch of the kernel matrix and use it in lieu of the full matrix in the machine learning task of interest. Here, we describe a version of this approach that comes with running time guarantees as well as improved guarantees on its statistical performance. By extending the notion of statistical leverage scores to the setting of kernel ridge regression, we are able to identify a sampling distribution that reduces the size of the sketch (i.e., the required number of columns to be sampled) to the effective dimensionality of the problem. This latter quantity is often much smaller than previous bounds that depend on the maximal degrees of freedom. We give an empirical evidence supporting this fact. Our second contribution is to present a fast algorithm to quickly compute coarse approximations to these scores in time linear in the number of samples. More precisely, the running time of the algorithm is O(np 2 ) with p only depending on the trace of the kernel matrix and the regularization parameter. This is obtained via a variant of squared length sampling that we adapt to the kernel setting. Lastly, we discuss how this new notion of the leverage of a data point captures a fine notion of the difficulty of the learning problem.

This paper extends on previous work relating to the Nystrom method of low rank approximation of SPSD (e.g. kernel) matrices, showing that tighter bounds can be derived that the sharp analysis of Bach using a non-uniform sampling strategy that attempts to control the bias. The idea of non-uniform sampling for the Nystrom approximation is not itself new, but the analysis goes further than this, and holds for any sketching matrix that meets certain conditions. This, combined with the resultant bounds, is a significant contribution. Comments/criticisms: - the title is "Fast Randomized Kernel Methods ..." whereas currently the analysis only holds for Kernel Ridge Regression. **Note that the title was changed for final submission** - it seems that this method will not work for any normalised kernel matrix (i.e. where f(x) = \phi(x)/ \left| \phi(x) \right|), since the p_i's will be uniform. For example, the RBF kernel will always have 1's on the diagonal, so K_{ii} / TR(K) will just be 1/n? - in the first experiment, what is the MSE when using full KRR on this dataset? does the lambda ridge method converge to this error? - in the second experiment (Table 1), why haven't MSE's been reported? A comparison with existing methods should also be done here.Specific comments: - for ref [1] it would be better to refer to the published ICML paper than the unpublished arxiv paper unless there's a good reason not to (http://machinelearning.wustl.edu/mlpapers/bibtex/icml2013_gittens13) - for ref [5] same as above (http://jmlr.org/proceedings/papers/v32/ma14.html) - L67, L256 the authors refer to Bach's d_{mof} as "maximal degrees of freedom", but I believe Bach called this "marginal degrees of freedom" - eq 4 (and later) the first term is given as bias(K)^2 whereas Bach denoted this same term as simply bias(K) - is there a reason for changing this? - L158The operator  is used for the first time. I presume you are referring to the linear matrix inequality X \succeq Y \Longleftrightarrow \lambda_{min}(X-Y)\geq 0, but I'm not sure all readers will be aware of this - L162 derived of => derived from - Theorem 2. State what t is (i.e. t \in (0, 1)?) - L302-305 Section 3.5. The last sentence in the first paragraph is unclear and too long, please reword this - L373 experiments -> experiment - Figure 1. These figures are too small to be readable. I realise space is an issue but please make these more readable if possible. - L398 haven't we known there existence => had we not known of their existence - L399 determine \lambda => determine \lambda and \sigma - Table 1. \sigma was used earlier in the paper to refer to eigenvalues, now it refers to the RBF parameter - could a different symbol be used? - throughout the paper (and references inverted commas are the wrong way round, caused by using the double quote " character instead of two single quotes '' - In the references the capitalisation is wrong in many cases: conference/journal papers should have title case (except for proper nouns and abbreviations), whereas books should have every word capitalized. Nystrom should appear as Nystr\"om. Journal papers should have issue numbers and page numbers.


Online Learning for Adversaries with Memory: Price of Past Mistakes
The framework of online learning with memory naturally captures learning problems with temporal effects, and was previously studied for the experts setting. In this work we extend the notion of learning with memory to the general Online Convex Optimization (OCO) framework, and present two algorithms that attain low regret. The first algorithm applies to Lipschitz continuous loss functions, obtaining optimal regret bounds for both convex and strongly convex losses. The second algorithm attains the optimal regret bounds and applies more broadly to convex losses without requiring Lipschitz continuity, yet is more complicated to implement. We complement the theoretical results with two applications: statistical arbitrage in finance, and multi-step ahead prediction in statistics.

The paper extends the notion of online learning with adversarial memory to the OCO framework. The authors propose two efficient algorithms: first one applies to (strongly) convex and Lipschitz functions while the second applies to convex functions, without requiring Lipschitz property. The authors complement their results with experiments.Originality- I will not question this aspect of the paper. The authors mostly adapt existing theory in online learning but do it in a clever way, which, if correct (I will come to this later) is worthy of acceptance. Significance- I believe the idea will be interesting to the online learning community. Clarity- I have some minor issues regarding the clarity of the paper. I will list them so that (hopefully) the authors find it easier to read. 1. A concrete example of a function which calculates loss based on last $m$ moves of learner would have been helpful in the introduction.The authors simply say "However, in coding, compression, portfolio selection and more, the adversary is not completely memoryless". While I am sure I can find examples in the references, the introduction should be more enticing and the paper self-containing. 2. Line 156- It is mentioned that f_t is convex if $\tilde{f}(x)= f(x,x,...,x)$ is convex. Considering $f_t$ is a multivariate function, should it not be that $f_t$ being convex naturally mean that it is jointly convex in all $m$ arguments? Otherwise, why claim $f_t$ to be convex? 3. Theorem 3.1- The regret is for Lipschitz continuous function with constant L (line 176). However, L does not even appear in the regret equation. Looking at the proof, it seems L has somehow been absorbed in \lambda and \sigma. This is very confusing to someone reading the main paper.And in line 705, why is the Lipschitz constant $\sqrt(\lambda) \sigma$? 4. Though I understand the more involved work is for (non)-Lipschitz functions, Algorithm 1. and its' proof is extremely simple when regularizer is $\ell_2$ norm (basically OGD). In fact, in OGD, two consecutive iterates $x_{t+1}$ and $x_t$ are close as long as the gradient is controlled (it is when the function is Lipschitz). Maybe the analysis of Alg.1 can be moved to the supplement with more discussion about Alg 2. in the main paper. 5. Why are \hat{g}_t exp-concave with \eta/4G^2 (line 893)? I understand its exp-concave but why with factor \eta/4G^2?Quality- My major concern is with the proof of one lemma. I am enlisting my concern here: 1. Lemma D.3- The crucial part of Alg.2 is the bound on the number of switches (As formally stated in Lemma D.4).So effectively, the proof goes through the following line of argument: Alg.5 is modified in Alg.6, which changes the way $x_{t+1}$ is sampled. Through a series of arguments, it comes to the inequality in Lemma D.2. Then Theorem 4.1 follows by using a bound on number of switches.My concern is with Lemma D.3. First of all, both Alg.2 and Alg.6 are being applied to exp-concave function $\hat{g}_t$ (Change the function $\ell_t$ to $\hat{g}_t$ in Alg.6). I dont think it is correct to claim Alg.2 is being applied to function $g_t$. The loss is being calculated in terms of $g_t$ but the update is based on $\hat{g}_t$, since $\hat{g}_t$ is exp-concave. So the only difference between Alg.2 and Alg.6 is how the sequence $x_t$ is being generated. Crucially, in Alg.6, $x_t$ is being sampled from $p_t(x)$. In Alg.2 (using the same exp-concave functions), $x_t$ is being generated from $p_t(x)$ w.p.1- (w_t(x_{t-1})/w_{t-1}(x_{t-1})) and $x_t=x_{t-1}$with complementary probability. Since $p_t$ is same density function in Alg.2 and Alg.6, how can the two sequences comefrom same distribution? I find that very much counter-intuitive, and this lies in the heart of analysis of Alg.2. Moreover, when i tried to go through the proof of Lemma D.3, i got stuck at the first line. Why is Line 947 true?Overall, I am not sure about the veracity of the claim. I am of course not claiming it is wrong but I think its crucial to verify the proof. I would request other reviewers/ PC members to check the proof of Lemma D.3 at least.Update: After discussion, I have understood the proof of Lemma D.3. I have updated by rating. Assuming that the results have not appeared anywhere before, the paper should be accepted


Adaptive Stochastic Optimization: From Sets to Paths
Adaptive stochastic optimization (ASO) optimizes an objective function adaptively under uncertainty. It plays a crucial role in planning and learning under uncertainty, but is, unfortunately, computationally intractable in general. This paper introduces two conditions on the objective function, the marginal likelihood rate bound and the marginal likelihood bound, which, together with pointwise submodularity, enable efficient approximate solution of ASO. Several interesting classes of functions satisfy these conditions naturally, e.g., the version space reduction function for hypothesis learning. We describe Recursive Adaptive Coverage, a new ASO algorithm that exploits these conditions, and apply the algorithm to two robot planning tasks under uncertainty. In contrast to the earlier submodular optimization approach, our algorithm applies to ASO over both sets and paths.

*Summary*This paper studies the adaptive stochastic optimization on paths, by imposing some constraints (namely the MLRB and the MLB conditions) on the relationship between the probability of a realization and its utility. While the notion of *adaptive submodularity* states that in expectation, no item will "surprise" us given a longer history, the MLRB / MLB conditions in this paper roughly characterizes the fact that such "surprise" only happens with small probability (i.e., realization with small probability has large utility). By leveraging such constraints, the authors propose a novel recursive algorithm with near-optimal guarantee on the expected cost needed to reach a certain coverage. Under the MLB conditions, the paper recovers the classical approximation result on bayesian pool-based active learning, up to a constant factor.*Quality*The paper is nicely written and well structured. Regarding the experimental results: by Theorem 4, it would be very interesting to see how the proposed RAC algorithm competes with existing algorithms *empirically* on the pool-based active learning problem (e.g., [3][5][7], or [6] under the bounded noise setting), given the fact that the version space reduction function and the Gibbs error function both satisfy MLRB condition, pointwise submodularity condition, and adaptive submodularity condition. Having these results will certainly make the paper stronger.*Clarity*Most of the paper is clear. It would be helpful if the authors elaborate more on the intuitive explanations of the two conditions: why are those two conditions more natural than the adaptive submodularity condition for adaptive optimization problems on paths?*Originality and Significance*To the best of my knowledge, the technical contribution of this contribution is novel. It provides interesting insights to developing tractable algorithms for the adaptive stochastic optimization problem.


VISALOGY: Answering Visual Analogy Questions
In this paper, we study the problem of answering visual analogy questions. These questions take the form of image A is to image B as image C is to what. Answering these questions entails discovering the mapping from image A to image B and then extending the mapping to image C and searching for the image D such that the relation from A to B holds for C to D. We pose this problem as learning an embedding that encourages pairs of analogous images with similar transformations to be close together using convolutional neural networks with a quadruple Siamese architecture. We introduce a dataset of visual analogy questions in natural images, and show first results of its kind on solving analogy questions on natural images.

The paper presents a model for visual analogy retrieval. For a query image one wants to find a second image which relates to the query in the same way as the relationship between a given pair of images. More precisely, an offset pair (i1, i2) defines the analogy. For a query i3 an image i4 is to be found such that i3 relates to i4 in the same way as i1 relates to i2.The model is formulated as an embedder, which embeds all four images ik -> xk (k = 1, ..., 4) such that (x4 - x3) = (x2 - x1). The embedder is a CNN based on AlexNet. The loss is a hinge loss. In addition, the authors present a dataset of visual analogy. The paper is well written and presents a well defined model which is empirically evaluate in two datasets. My main concern with this work is that the presented problem might be a bit contrived. The authors define an analogy as relationship between images defined by different properties of these images. The philosophy of analogies being a pair of two properties is incorporated in both of their datasets.So if an analogy is defined as pair of properties, I natural baseline would be to set up a classification problem for these properties. Then if analogy is defined by the properties of two images, then one can apply the trained classifier to estimate these properties. A second classifier would determine the class of the query image. Then the analogous image would be one of the class of the query and the missing of the two properties. Such a baseline is based on two classifiers and would be simple to set up.


A Complete Recipe for Stochastic Gradient MCMC
Many recent Markov chain Monte Carlo (MCMC) samplers leverage continuous dynamics to define a transition kernel that efficiently explores a target distribution. In tandem, a focus has been on devising scalable variants that subsample the data and use stochastic gradients in place of full-data gradients in the dynamic simulations. However, such stochastic gradient MCMC samplers have lagged behind their full-data counterparts in terms of the complexity of dynamics considered since proving convergence in the presence of the stochastic gradient noise is nontrivial. Even with simple dynamics, significant physical intuition is often required to modify the dynamical system to account for the stochastic gradient noise. In this paper, we provide a general recipe for constructing MCMC samplers-including stochastic gradient versions-based on continuous Markov processes specified via two matrices. We constructively prove that the framework is complete. That is, any continuous Markov process that provides samples from the target distribution can be written in our framework. We show how previous continuous-dynamic samplers can be trivially "reinvented" in our framework, avoiding the complicated sampler-specific proofs. We likewise use our recipe to straightforwardly propose a new state-adaptive sampler: stochastic gradient Riemann Hamiltonian Monte Carlo (SGRHMC). Our experiments on simulated data and a streaming Wikipedia analysis demonstrate that the proposed SGRHMC sampler inherits the benefits of Riemann HMC, with the scalability of stochastic gradient methods.

The authors propose a general framework for designing new MCMC samplers, including methods that use stochastic gradients. Their approach is to define a stochastic dynamical system whose stationary distribution is the target distribution from which we want to sample from. The stochastic dynamical system is represented through a stochastic differential equation that is simulated through an epsilon-discretization approach. As the step-size parameter epsilon goes to zero, the bias in the simulation vanishes. The proposed approach can handle stochastic approximations obtained by sub-sampling the data in mini-batches. For this, the additional noise produced by the sub-sampling process is removed from the noise used to simulate the SDE. The proposed framework constructs the SDE using two state dependent matrices D(z) and Q(z) and by modifying these matrices the authors can explore the space of MCMC methods. The authors proof that their approach samples from the correct stationary distribution as the step size epsilon tends to zero. The authors also proof that any continuous Markov process with a particular stationary distribution can be represented in their framework and consequently, such framework is complete. After this, the authors show how several already existing samplers can be represented in their framework: Hamiltonian Monte Carlo, Stochastic Gradient Hamiltonian Monte Carlo, Stochastic Gradient Langevin Dynamics, Stochastic Gradient Riemannian Langevin Dynamics and Stochastic Nose-Hoover Thermostast. Finally, the authors proposed based on their framework a new sampler technique called Stochastic Gradient Riemann Hamiltonian Monte Carlo (SGRHMC). This method uses the Ficsher information matrix to precondition the dynamics through the matrices D(z) and Q(z). The SGRHMC method is shown to provide very good results on experiments with simulated data and streeming data extracted from Wikipedia.Quality:The paper is technically very sound, with theorems showing the applicability of the framework to any target distribution and its completeness in terms of how many continuous Markov processes it can represent.Clarity:The paper is clearly written and well organized. A few things could be improved for clarity: How do you construct in practice the matrix \hat{B}_t? When you substract \hat{B}_t to eliminate the stochastic gradient noise term, what happens if you get a covariance matrix that is not possitive definite? Perhaps this is not possible if \epsilon is small. In line 232 the minus sign in Q should probably be changed from the upper right identity matrix to the lower left one.Originality:The proposed framework is original. It allows the authors to re-design existing MCMC samples and propose new ones with better properties. The author's contribution is significantly different from previous work.Significance:This work is highly significant, specifying a principled way to design new scalable MCMC samples that work by sub-sampling the data and are able to sample from the correct target distribution in the limit of the step size tending to zero.


Bayesian Dark Knowledge
We consider the problem of Bayesian parameter estimation for deep neural networks, which is important in problem settings where we may have little data, and/ or where we need accurate posterior predictive densities p(y|x, D), e.g., for applications involving bandits or active learning. One simple approach to this is to use online Monte Carlo methods, such as SGLD (stochastic gradient Langevin dynamics). Unfortunately, such a method needs to store many copies of the parameters (which wastes memory), and needs to make predictions using many versions of the model (which wastes time). We describe a method for "distilling" a Monte Carlo approximation to the posterior predictive density into a more compact form, namely a single deep neural network. We compare to two very recent approaches to Bayesian neural networks, namely an approach based on expectation propagation [HLA15] and an approach based on variational Bayes [BCKW15]. Our method performs better than both of these, is much simpler to implement, and uses less computation at test time.

Paper Title: Bayesian Dark KnowledgePaper Summary: This paper presents a method for approximately learning a Bayesian neural network model while avoiding major storage costs accumulated during training and computational costs during prediction. Typically, in Bayesian models, samples are generated, and a sample approximation to the posterior predictive distribution is formed. However, this requires storing many copies of the parameters of a model, which may require a great deal of storage/memory in high-parameter neural network models. Additionally, prediction on test data requires evaluation of all sampled models, which may be computationally costly. This paper aims to learn a model that makes (approximate) posterior-predictive-based predictions on test data without storing many copies of the parameter set. The method presented here accomplishes this by training a "student" model to approximate a Bayesian "teacher's" predictions---a procedure in the neural networks literature referred to as "distillation" (and also referred to in previous papers as "model compression"). However, the student and teacher are trained simultaneously in an online manner (so no large collection of samples are required to be stored, even during training). Experiments are shown on synthetic data (a toy 2D binary classification problem and a toy 1D regression problem), on classification for the MNIST dataset, and on regression for a housing dataset. Comments:- I feel that working to develop computational tools for practical Bayesian inference in neural networks is an important direction of research (in particular, tools that mitigate the issue of large storage/memory requirements when sampling in large-parameters Bayesian models), and I feel that this paper is making good steps in this direction.- One issue I have with the novelty of this paper is that the fundamental concept being developed here (student/teacher learning, or model distillation/compression) is not new --- the authors apply this existing idea in a new domain. However, I do feel the main novelty (and strength) of this paper is the clever way in which the algorithm carries out simultaneous online training of the teacher and student without requiring storage of samples. This is an online distillation/compression method in which the teacher is never "fully trained" (the teacher never actually provides full posterior-predictive "labels" for the student) and yet the student is still able to learn the "full trained" teacher's model.- One potential computational issue with the presented algorithm is that the student must be trained (to mimic the teacher), on the training dataset D', at every iteration during the MCMC algorithm. I believe this requires a great deal more training of the student than in typical distillation methods (which do not need to make many passes through the training dataset D'). This might be particularly problematic if D' is required to be large in order to achieve good performance. In general, I feel that there is not enough discussion about the training set D' (whether particular choices about D' affect the method's performance, and the particular details used in the experiments presented in this paper).- I feel that the experiments in this paper were not totally polished and thorough. The authors write that they could not enable a proper comparison of all methods on all datasets because (in part) "the open source code for the EP approach only supports regression" and "we did not get access to the code for the VB approach in time for us to compare to it". I appreciate the honesty here! However, it would be nice to complete these goals and polish the experiments section in this paper.


Consistent Multilabel Classification
Multilabel classification is rapidly developing as an important aspect of modern predictive modeling, motivating study of its theoretical aspects. To this end, we propose a framework for constructing and analyzing multilabel classification metrics which reveals novel results on a parametric form for population optimal classifiers, and additional insight into the role of label correlations. In particular, we show that for multilabel metrics constructed as instance-, micro-and macroaverages, the population optimal classifier can be decomposed into binary classifiers based on the marginal instance-conditional distribution of each label, with a weak association between labels via the threshold. Thus, our analysis extends the state of the art from a few known multilabel classification metrics such as Hamming loss, to a general framework applicable to many of the classification metrics in common use. Based on the population-optimal classifier, we propose a computationally efficient and general-purpose plug-in classification algorithm, and prove its consistency with respect to the metric of interest. Empirical results on synthetic and benchmark datasets are supportive of our theoretical findings.

SUMMARYThe authors develop a theoretically justified thresholding method for plug-in estimators in multilabel classification, based on analyzing the Bayes-optimal classifier for the given objectives. CONTRIBUTIONS: WHAT IS (ARE) THE CLAIMED CONTRIBUTION(S) OF THE PAPER? HOW SIGNIFICANT/IMPORTANT IS (ARE) THE CLAIMED CONTRIBUTION(S)?Compute the form of the Bayes-optimal multilabel for two families of objectives, and use it to construct a multilabel classification algorithms based on plug-in estimators with good empirical performance.The theoretical contributions add to the understanding of multilabel classification and the algorithms are a welcome finding. SUPPORTThe theoretical results are well supported.The algorithms are proposed in a principled way based on the theoretical results. Regarding Algorithm 1, I find the fact that the thresholds can be efficiently computed an interesting observation.The empirical evaluation seems more illustrative than descriptive of the proposed method, but validates the theoretical findings and provides favorable evidence about their impact. Nevertheless, the empirical results presented are informative because in the theoretical results the form of the bayes-optimal classifier is derived, while in practice the class of classifiers being used is more restricted than all measurable classifiers. Section 5 therefore bridges this gap. TECHNICAL QUALITYThe technical quality of the paper is good, but in this reviewer's opinion the text would benefit from some more discussion about certain topics, as described ahead.There seems to be a stronger connection between the problem studied with the chosen metrics and binary classification (rather than just binary classification being a special case of the multilabel classification). Evidence to that is the seemingly strong connection between the theoretical results presented and those of Koyejo et al. 2014 (see the detailed comments section for an example). The discussion of the theoretical results would benefit from a more careful exploration of these connections.Section 5 would be enriched by some conjectures regarding why Algorithm 1 was competitive with Macro-Thres with the macro-averaged metrics (in Table 3). It would also be interesting to see how close the thresholds of Algorithm 1 are to 1/2 where BR was competitive with the micro-averaged metrics (in Table 1). ORIGINALITYThe paper inserts itself well in the picture, and the only suggestion I can make here is for the authors to point out that their proof techniques are substantially similar to those used by Koyejo et al. 2014 -- in particular it would be helpful for anyone checking the proofs to understand where they differ and how the treatment of multilabel classification affects the proofs. CLARITYThe paper is clear and the text reads fluidly. There are a few missing articles in the text, but otherwise the work well written and well executed. FOR THE REBUTTALIf my remarks about the parallels between the authors' results and those of Koyejo et al. (2014) are pertinent, a brief comment from the authors on the topic would he useful. DETAILED COMMENTSI find the similarity between Theorem 2 in the paper and Theorem 2 by Koyejo et al. (2014) quite striking. Upon checking the proofs and seeing that they are essentially the same, I started to wonder the reason behind that. I would venture that the two theorems are, in fact, equivalent. To see that (with a finite input space) we can define $\mathcal{X}' \doteq \{ x_{ij} : x_i \in \mathcal{X}, j \in [M] \}$, $P(X' = x_{ij}) \doteq \frac{1}{M}P(X = x_i)$ and $P(Y' = 1 | X' = x_{ij}) \doteq P(Y_j = 1 | X = x_i)$. Defining the quantities $\mathrm{TP}'$, $\gamma'$ and $\mathcal{L}'$ accordingly, we get$\mathcal{L}'(f) = \frac{c_0 + c_1 \mathrm{TP}(f) + c_2 \gamma(f) }{d_0 + d_1 TP(f) + d_2 \gamma(f)}$. So we can get Theorem 2 in the paper from Theorem 2 by Koyejo et al. (2014). (Note the important influence of the decoupling over labels and the decoupling over inputs in the reduction.) A similar observation can also be made using Proposition 1 in the submission.On the other hand the macro-averaged metric seems to be solved by combining the Bayes-optimal classifiers for $M$ separate binary classification problems.I would also point out that (in my opinion) the threshold being shared over the labels is as impressive as its being shared over the inputs.I was checking the proof of Theorem 2, and I arrived at a different form for the gradient $$\frac{\partial}{\partial f(x_i)_m}\mathcal{L}(f) = \frac{\mu(x_i)(c_1 - d_1 \mathcal{L}(f))}{D_r(f)}\left( \eta_m(x_i) + \frac{c_2 - d_2 \mathcal{L}(f)}{c_1 - d_1 \mathcal{L}(f)} \right)$$ (similar to the proof of Theorem 2, but with $(c_1 - d_1 \mathcal{L}(f))$ in the numerator, not in the denominator), which can also be found in Koyejo et al. (2014). SUGGESTED FIXES[l:36] The authors need to make sure that the citation format conforms to the conference requirements. [l:78] Remove the period [l:115] $n$ should be $N$ here (make sure you check other occurrences of $n,N$) [l:133] "corresponding... corresponds" [l:159] "metric to the confusion" [l:162] $\Psi$ was overloaded a few times up to this point, and it confused me; the usage seems consistent afterwards, but I would suggest revisiting the previous definitions and adjusting them to minimize ambiguity. [l:239] "optimizing an EUM utility" [l:244] "optimal for the macro-averaged" [l:257] "learning a multi-label" [l:351] The end of a sentence seems to be missing here. [l:406] "metrics, an optimal"POST-REBUTTAL REMARKSAfter considering that the theoretical results are a special case of previously existing results, albeit interesting observations, they are not substantial contributions, from the theoretical point of view. However, they justify the proposed algorithm, which the experiments validate and which carry interesting insights.


Principal Geodesic Analysis for Probability Measures under the Optimal Transport Metric
Given a family of probability measures in P (X ), the space of probability measures on a Hilbert space X , our goal in this paper is to highlight one ore more curves in P (X ) that summarize efficiently that family. We propose to study this problem under the optimal transport (Wasserstein) geometry, using curves that are restricted to be geodesic segments under that metric. We show that concepts that play a key role in Euclidean PCA, such as data centering or orthogonality of principal directions, find a natural equivalent in the optimal transport geometry, using Wasserstein means and differential geometry. The implementation of these ideas is, however, computationally challenging. To achieve scalable algorithms that can handle thousands of measures, we propose to use a relaxed definition for geodesics and regularized optimal transport distances. The interest of our approach is demonstrated on images seen either as shapes or color histograms.

In this paper, the authors proposed a novel algorithm to compute principal geodesics in the Wasserstein space. Experiments in simulated and real datasets show the promising performance. Essentially, the PCA is generalized into nonlinear PCA when such estimation is performed under the Wasserstein space, which is a high-dimensional nonlinear space. However, it is not easy to see such nonlinearity is well described through observing several experimental results reported in this paper. Specifically, 1) in Fig, 1, 2 and 3,I guess these principal Geodesics should be principal components whereas in my opinion, the real principal Geodesics should be principal curves in one-dimensional sense or principal manifolds in two or higher dimensional sense.Further, since the authors do not explain what the points are,I have no ideas on how the algorithms run for the data points and whether the results are the best ones. 2) Fig. 4 and Fig. 5 should be used to discover one intrinsic dimension or the first principal Geodesic. Therefore, if the intrinsic dimensions are greater than 3, maybe the algorithms will not work.I think the authors should discuss the limitation of their proposed algorithms related to the dimension of intrinsic variables. 3) Orthogonality: I agree that the authors are right that in supplementary material, Figure 1 indeed discovered two intrinsic variables: scale and translation. However, for other figures, it is not easy to find such a nonlinear orthogonality. Especially for other images, I have difficult to figure out the "orthogonality" from them. Therefore, I think that the authors should compare their proposed algorithm with not only PCA but also other nonlinear PCA algorithms so that we can see how much the algorithms improve the estimation of principal Geodesics based on different datasets.


Learning visual biases from human imagination
Although the human visual system can recognize many concepts under challenging conditions, it still has some biases. In this paper, we investigate whether we can extract these biases and transfer them into a machine recognition system. We introduce a novel method that, inspired by well-known tools in human psychophysics, estimates the biases that the human visual system might use for recognition, but in computer vision feature spaces. Our experiments are surprising, and suggest that classifiers from the human visual system can be transferred into a machine with some success. Since these classifiers seem to capture favorable biases in the human visual system, we further present an SVM formulation that constrains the orientation of the SVM hyperplane to agree with the bias from human visual system. Our results suggest that transferring this human bias into machines may help object recognition systems generalize across datasets and perform better when very little training data is available.

Summary: This paper draws inspiration from work on psychophysics on classification images.Large-scale human experiments were run, where people were asked to classify images generated from random noise (randomly generated by inverting HOG or CNN feature spaces to more closely approximate the distribution of natural images).The results were used to 1) visualize human perception of different classes, 2) see how well classifiers trained on datasets of random noise would work on real images, and 3) use the results as an additional source of information to regularize classifiers trained on a small number of images.Quality:This is a very unusual paper.It is overall a high quality and well written paper where interesting and novel experiments were carried out; however it is unclear if the results or methods of the paper are of practical value.Clarity:The paper is well written and clear.Originality:This is a really creative paper with an interesting and novel idea.It is nice that the authors drew in research from other disciplines (psychophysics) that most people in machine learning are unfamiliar with.Significance:I think the main value of this paper is that it will inspire readers to think about philosophical issues like what is dataset bias?, does the definition of a class come from human imagination or is inherent to the world?, how does one classify images that lie outside the distribution of natural images?I think in general people will find it fun and interesting to see how people perceive random images, as well as some of the qualitative results that people from India had different perceptions than people from the US.At the same time, the claims of the paper are controversial (see points 1 & 2 below) and the methods realistically have little practical value (see point 3 below).1) Does the proposed method alleviate problems of dataset bias? I don't think so.I think dataset bias emerges if one trains from datasets/images that come from a different distribution than natural data or problems we ultimately would like to solve (admittedly this is a fuzzy and probably misguided definition). Training a classifier (or generating a classification image) by having people label random white noise in pixel space is training from an extremely different (and also biased) synthetic distribution.Training from random noise in HOGgles space is similarly another type of biased synthetic dataset.I think it is an interesting finding to see how people classify images outside the domain of natural images, but this is an entirely different issue than dataset bias.A machine learning algorithm has biases because of the data that was used to train it as well as limitations of the particular learning model.Humans have biases due to their personal experiences as well as their learning models (ie., architecture of the brain).An algorithm trained from human labels will inherently (attempt to) learn to capture the biases of its human labelers.This isn't altered by using random images as opposed to natural images.Rather, using random images exacerbates the problem by using an even more unnatural distribution of data.2) Does the notion of classification images from psychophysics literature offer some new insight into human perception or dataset bias? I don't think so, or at least the paper doesn't go far enough in explaining this concept to communicate this to a NIPS audience that is unfamiliar with psychophysics research (such as myself).The 2 technical differences in the proposed classification image-based approach compared to methods used in machine learning are a) having people label randomly generated training data instead of natural images, and b) the method of generating classification images to visualize human classifiers.Looking at cited literature [23,9] classification image literature are making a lot of assumptions about human perception (e.g., a linear separator in feature space) and these assumptions are similar to some models used in machine learning (i.e., similar to a simplified version of Fisher's linear discriminant).I am unable to grasp why using a random noise training set is desirable or necessary, or if this is just a method of choice to acquire a dataset.3) The proposed method of regularizing classifiers trained from a small number of images with classification images is creative but impractical.It required tons of human annotation labelling random images for very small improvements in performance.


Tree-Guided MCMC Inference for Normalized Random Measure Mixture Models
Normalized random measures (NRMs) provide a broad class of discrete random measures that are often used as priors for Bayesian nonparametric models. Dirichlet process is a well-known example of NRMs. Most of posterior inference methods for NRM mixture models rely on MCMC methods since they are easy to implement and their convergence is well studied. However, MCMC often suffers from slow convergence when the acceptance rate is low. Tree-based inference is an alternative deterministic posterior inference method, where Bayesian hierarchical clustering (BHC) or incremental Bayesian hierarchical clustering (IBHC) have been developed for DP or NRM mixture (NRMM) models, respectively. Although IBHC is a promising method for posterior inference for NRMM models due to its efficiency and applicability to online inference, its convergence is not guaranteed since it uses heuristics that simply selects the best solution after multiple trials are made. In this paper, we present a hybrid inference algorithm for NRMM models, which combines the merits of both MCMC and IBHC. Trees built by IBHC outlines partitions of data, which guides Metropolis-Hastings procedure to employ appropriate proposals. Inheriting the nature of MCMC, our tree-guided MCMC (tgMCMC) is guaranteed to converge, and enjoys the fast convergence thanks to the effective proposals guided by trees. Experiments on both synthetic and realworld datasets demonstrate the benefit of our method.

The paper proposes an efficient MCMC inference for normalized random measure mixture models. Borrowing the idea of hierarchical Bayesian clustering,the inference algorithm uses tree structures to represent cluster and develops stochastic tree operations to generate proposals; it inherits the benefit of incremental Bayesian clustering algorithm (ICBH)---the operation over trees is efficient and the tree-guided proposals are empirically effective; it overcomes the deficiency of IBHC by developing the inference rigorously in MCMC framework: the transition probability is clearly defined and the convergence is theoretically guaranteed. The paper is well written and the work is solid. The only issue to me is that on real dataset, the proposed algorithm shows slower mixing rate than the competing methods, which is a little disappointing; maybe the authors should test more real datasets for confirmation.


GAP Safe screening rules for sparse multi-task and multi-class models
High dimensional regression benefits from sparsity promoting regularizations. Screening rules leverage the known sparsity of the solution by ignoring some variables in the optimization, hence speeding up solvers. When the procedure is proven not to discard features wrongly the rules are said to be safe. In this paper we derive new safe rules for generalized linear models regularized with 1 and 1 { 2 norms. The rules are based on duality gap computations and spherical safe regions whose diameters converge to zero. This allows to discard safely more variables, in particular for low regularization parameters. The GAP Safe rule can cope with any iterative solver and we illustrate its performance on coordinate descent for multi-task Lasso, binary and multinomial logistic regression, demonstrating significant speed ups on all tested datasets with respect to previous safe rules.

Summary:This paper presents an unprecedentedly fast method for eliminating variables during iterations of multi-task group lasso that is provably safe, meaning that all variables that are eliminated would ultimately obtain zero weights when running vanilla group lasso.This method is iterative; as the primal solver converges, it eliminates an increasing number of variables.The authors compare their method to previous methods and demonstrate that all previous methods are either unsafe or are substantially slower for small duality gaps.The authors describe how their method should be applied to specific cases of group lasso, including l1 and l1/l2 regularized logistic regression, and present real applications where their method achieves a substantial speed-up over vanilla group lasso and an existing method for small duality gap thresholds.General suggestions: 1. The primary situation where this method obtains substantial speed improvements over alternatives is where the duality gap threshold is extremely small.To make the importance of the work more convincing, the authors need to demonstrate that this is a common situation or at least a situation that occurs in some important group lasso settings.If the duality gap must be small in one of the real examples the author use in Section 4, then stating this and providing a citation for it would be sufficient. 2. Some of the derivations skipped many steps.The paper would be much easier to follow if these derivations were fleshed out in more detail, either in the main text or in the Supplement.I have noted the places where this was the case in "Specific comments and suggestions." 3. There are many grammar and word choice errors, including in the statements of many of the propositions.To make the paper sufficiently clear for readers to grasp what the authors have accomplished, the authors need to fix these errors.I have pointed out the most problematic ones in "Specific comments and suggestions."Specific comments and suggestions, main text: 1. In the Introduction, the authors discuss the lasso problem.In Section 2, they discuss the group lasso problem.They should explicitly state that the lasso is a specific case of the group lasso and maybe re-word the introduction to discuss the group lasso problem. 2. There is a missing citation in Remark 1 in Section 2.1. 3. The authors should cite Fermat's condition at the beginning of Section 2.2. 4. The authors should either derive the dual formulation of the group lasso that they present in Section 2.2 in the Supplement or cite the paper in which it was derived. 5. In the first "if and only if" statement in Section 2.3, I am convinced that 0 being a primal solution implies the next statement, but it is not obvious why the converse is true.The authors should justify the converse in more detail. 6. The authors should add a conclusion sentence to section 2.3 that summarizes what they have shown. 7. In the beginnings of the subsections of Section 2.6, the word "Remind" should be replaced with "Remember." 8. It is not obvious why the sentence "Since theta^(gamma) maximizes D_gamma..." near the beginning of Section 2.6 is true, so the authors should justify this in greater detail. 9. Remark 4, which refers to equation (15), should be moved after equation (15).It is also not obvious why the first sentence is true, so the authors should justify it in greater detail. 10. When summarizing the Dynamic GAP Safe rule in Section 2.6, the authors state that dynamic safe screening rules are "the most efficient."They either need to rigorously prove that making more efficient rules would be impossible or replace the phrase "the most efficient" with something like "more efficient than existing methods in practice." 11. The way that Proposition 1 in Section 2.6 is worded is very confusing, so the authors should re-word it. 12. In Remark 7, the authors say that the active set obtained by GAP safe rules allow the solver to "identify relevant features earlier."They should say what method they are comparing to.Do they mean "earlier than previous methods would?" 13. In Section 3, the authors frequently use a variable z.They should define z at the beginning of the section. 14. The authors should add conclusion sentences to Sections 3.1 and 3.2 that summarize what they have shown. 15. I am not sure if the formulation of the logistic regression in (12) in Section 3.3 is correct.I think that the y_i should be in front of the parentheses and that there should be a term for the other class. 16. I am not sure if the formulation of the multinomial logistic regression in (14) is correct.I think that the Y_i should be in front of the second summation. 17. It is not obvious why the final sentence of Section 3.3 is true, so the authors should justify it in greater detail. 18. It is not obvious why the final sentence of Section 3.4 before Remark 9 is true, so the authors should justify it in greater detail. 19. Table 1 in Section 3 should be referred to somewhere in the main text. 20. In Section 4, the authors describe how they implemented the GAP Safe rule.They should make the code publicly available so that others can easily use their algorithm when solving group lasso problems. 21. In each of the applications the authors describe in Section 4, the author should explain why the optimization problem they are using is the right way to solve the problems or cite papers that have these explanations. 22. The heatmaps in Figures 1-3 are confusing.They might be clearer if the axes went in the opposite directions.The authors should also explicitly label the color bars in the figures. 23. In Section 4 and elsewhere, the authors sometimes call their method "GAP SAFE (Sphere)" and sometimes call it "GAP Safe (dynamic)."The authors should either always use the same terminology or, if these methods are not the same, explicitly define them and state the difference. 24. In Section 4, the authors compare to different existing methods for different datasets.The authors should explain why they do not compare to every method for every dataset. 25. In Section 4.2, the authors should summarize the results presented in Figure 2 in the text. 26. In Section 4.2, the authors should describe the "safe rule close in spirit to the Slores itself" or refer to the description in the Supplement. 27. The sentence in Section 5 starting with "Extensions to other type" is confusing, so the authors should re-word it.Specific comments and suggestions, supplement: 1. The phrasing of Proposition 2 in Section A.1 is a little confusing, so the authors should re-word it. 2. In the second paragraph in Section B, the authors should replace "sure" with "safe." 3. It is not obvious why the last sentence of Section B is true, so the authors should explain it in greater detail. 4. The phrasing of Proposition 4 in Section C.1 is a little confusing, so the authors should re-word it. 5. In Section C.1, it is not obvious why the choice for alpha[theta] is optimal when r_gamma_t-1 = 0, so the authors should explain why this is true in more detail. 6. The phrasing of Proposition 5 in Section C.2 is a little confusing, so the authors should re-word it. 7. When deriving the bound on |alpha[theta']- alpha[theta]| in Section C.2, it is not obvious why the second and third inequality are true, so the authors should provide additional steps. 8. In section C.2, it was not obvious why the final inequality is true, so the authors should provide additional steps.


Variational Consensus Monte Carlo
Practitioners of Bayesian statistics have long depended on Markov chain Monte Carlo (MCMC) to obtain samples from intractable posterior distributions. Unfortunately, MCMC algorithms are typically serial, and do not scale to the large datasets typical of modern machine learning. The recently proposed consensus Monte Carlo algorithm removes this limitation by partitioning the data and drawing samples conditional on each partition in parallel [22]. A fixed aggregation function then combines these samples, yielding approximate posterior samples. We introduce variational consensus Monte Carlo (VCMC), a variational Bayes algorithm that optimizes over aggregation functions to obtain samples from a distribution that better approximates the target. The resulting objective contains an intractable entropy term; we therefore derive a relaxation of the objective and show that the relaxed problem is blockwise concave under mild conditions. We illustrate the advantages of our algorithm on three inference tasks from the literature, demonstrating both the superior quality of the posterior approximation and the moderate overhead of the optimization step. Our algorithm achieves a relative error reduction (measured against serial MCMC) of up to 39% compared to consensus Monte Carlo on the task of estimating 300-dimensional probit regression parameter expectations; similarly, it achieves an error reduction of 92% on the task of estimating cluster comembership probabilities in a Gaussian mixture model with 8 components in 8 dimensions. Furthermore, these gains come at moderate cost compared to the runtime of serial MCMC-achieving near-ideal speedup in some instances.

Paper Title: Variational Consensus Monte CarloPaper Summary: This paper presents a new method for aggregating samples in a low-communication MCMC setting. In this setting, a large dataset is partitioned over multiple cores, MCMC is performed on a subset of data on each core (technically, samples are drawn from a subposterior distribution on each core), and the resulting samples are aggregated with some combination function to produce samples from the full posterior distribution. Instead of developing a fixed combination function, as previous methods have done, this paper generalizes the Consensus Monte Carlo [Scott et al., 2013] combination function to define a family of combination functions, and performs variational inference over this family to find the function that yields samples from the best approximation (within the family) to the full posterior distribution. One advantage of this method is that it may allow for easier combination of more sophisticated mathematical objects (instead of simply vectors) due to the generalized form of the aggregation. Experiments are shown on a Bayesian probit regression model, a normal-inverse Wishart model, and a mixture of Gaussians. Comments:- I feel that the goal of developing scalable methods for Bayesian inference that maintain a good approximation to the posterior distribution is important, and that this paper takes a good step towards this end. I also feel that this the paper provides a clever way of viewing an existing (low-communication) parallel MCMC strategy in terms of optimizing over a variational objective (another, typically separate, strategy in approximate Bayesian inference).- The primary goal of these low-communication parallel MCMC methods is to reduce the time of inference (while maintaining a good approximation of the posterior distribution), particularly relative to communication-based parallel MCMC methods. This paper does a good job of presenting a method that provides a good approximation of the posterior distribution (compared with existing methods), though there are relatively few experiments verifying that this method maintains the increased speed benefits of low-communication MCMC (relative to existing low-communication parallel MCMC methods, and communication-based parallel MCMC methods). This paper does indeed show comparison against speedups attained by the CMC method in one model, though a more thorough investigation would be nicer.- I wonder how the presented VCMC method would compare (in terms of inference speed and error) against serial methods for variational inference. VCMC was presented as a "variational Bayes algorithm", though all comparisons were against MCMC methods, and not variational Bayes methods. At its heart, this method is optimizing an approximate posterior (formed via distributed sampling), so it seems natural to compare against variational inference methods. This is particularly important given the recent popularity of scalable variational inference methods (such as stochastic gradient variational inference). Furthermore, there have recently been a few presented methods for low-communication parallel variational inference, which would make good experimental comparisons---see the following two references: (1) Broderick, Tamara, et al. "Streaming variational bayes." Advances in Neural Information Processing Systems. 2013. (2) Campbell, Trevor, and Jonathan P. How. "Approximate Decentralized Bayesian Inference." UAI. 2014.- One downside of this method is that it has little ability to make any guarantees about the correctness of the final aggregated samples (which some of the current scalable MCMC methods attempt to do); it simply generates samples using the the best combination function chosen out of a pre-specified family of functions. On the other hand, I suppose this isn't much worse than most variational inference methods (which simply choose the best posterior approximation from a family of distributions). Furthermore, this method should, in general, produce better results than CMC method [Scott et al., 2013] (assuming the weighted-average combination function is in the pre-specified family of functions).- A few theoretical results were shown in this paper (blockwise concavity under certain conditions, and consequences of this). It would be nice to include more discussion regarding the purpose of these results, and why it is beneficial to prove them (perhaps also mentioning what types of concavity results are typically shown in variational inference literature); currently, they are added in the paper without much comment as to why.


Efficient and Parsimonious Agnostic Active Learning
We develop a new active learning algorithm for the streaming setting satisfying three important properties: 1) It provably works for any classifier representation and classification problem including those with severe noise. 2) It is efficiently implementable with an ERM oracle. 3) It is more aggressive than all previous approaches satisfying 1 and 2. To do this, we create an algorithm based on a newly defined optimization problem and analyze it. We also conduct the first experimental analysis of all efficient agnostic active learning algorithms, evaluating their strengths and weaknesses in different settings.

The paper presents an active learning algorithm dedicated to the streaming setting: example per example, the algorithm decides whether to ask for a label or not. If good classifiers disagree on the example X, the proposed algorithm queries its label with a given probability given by the level of disagreement. If all good classifiers agree on X, the algorithm automatically labels the example with the predicted value.I confess that I am unfamiliar with most of the current work in active learning, and that I did not verify the validity of the theorems (which proofs are provided in supplementary material). Nevertheless, it appears to be a substantial piece of work, which reunites several previous ideas and push them forward. The good behavior of the proposed algorithm is theoretically justified by generalization guarantees and a bound on the number of queries. Moreover, the empirical experiments corroborate the effectiveness of the approach. The length and content of supplementary material is impressive (it contains 27 pages of non-trivial proofs, details about the experiments and supplementary results). The authors may have faced dilemmas when choosing information to give in the main paper (it is somewhat strange that the final learning algorithm is only explained in an appendix). This may explain the only complain I have about the paper: As a reader, I felt that the ideas were exposed in an expeditious manner. It would be appreciated to introduce some concepts that might be unknown to a reader not familiar with previous works in this area (like your humble reviewer!). Specifically, the introduction should define the exact meaning that the following expressions have in the specific studied framework: "agnostic active learning", "classifier representation", "oracle". Similarly, Section 5 should precise the meaning of "Oracular-CAL". Also, Algorithm 1 and Optimization Problem (OP) rely on many constant values, intervening in relatively complex expressions. To really understand their origin and meaning, the reader must go throughout the long proofs provided as supplementary material. I would appreciate being given an intuition of the meaning of each expression. Moreover, it would do no wrong to sometimes recall where expressions are defined. Indeed, the definitions are scattered. For example, D_j at Line 155 is defined inside Algorithm 1, \mathcal{D} and \mathcal{I}_h^m in Algorithm 2 (page 5) are respectively defined at Line 270 and by Equation (8) (page 6).Minor typo in Appendix F : Theorem 2 => Theorem 3 (two times).


Infinite Factorial Dynamical Model
We propose the infinite factorial dynamic model (iFDM), a general Bayesian nonparametric model for source separation. Our model builds on the Markov Indian buffet process to consider a potentially unbounded number of hidden Markov chains (sources) that evolve independently according to some dynamics, in which the state space can be either discrete or continuous. For posterior inference, we develop an algorithm based on particle Gibbs with ancestor sampling that can be efficiently applied to a wide range of source separation problems. We evaluate the performance of our iFDM on four well-known applications: multitarget tracking, cocktail party, power disaggregation, and multiuser detection. Our experimental results show that our approach for source separation does not only outperform previous approaches, but it can also handle problems that were computationally intractable for existing approaches.

The authors introduce a new Bayesian nonparametric model, more specifically an infinite factorial dynamic model (iFDM). This model is based on the Markov Indian buffet process and it allows for a possibly unbounded number of hidden Markov chains, each of which evolve independently according to some dynamics. The resulting inference problem is solved using particle Gibbs with ancestor sampling. The application in mind is that of source separation and the approach is evaluated on four examples; multitarget tracking, cocktail party, power disaggregation, and multiuser detection. The iFDM contains several interesting simpler models as special cases. Why is the state of the underlying Markov chain restricted to be of order one? There is a comment in the multitarget tracking example that the existing BNP approaches cannot handle this. What is the problem exactly?You are using a lot of particles, do you experience problems with degeneracy?You can probably improve the performance quite a lot by making use of a fully adapted filter, rather than the bootstrap as you are currently doing. Since your models are analytically tractable I think this should be quite straightforward to derive. More specifically this boils down to changing equation (4) and possibly also the resampling step. An early and basic introduction is provided in (Section II.D)http://www.stats.ox.ac.uk/~doucet/doucet_godsill_andrieu_sequentialmontecarloforbayesfiltering.pdf where it is referred to as the optimal proposal. If you google ``fully adapted particle filters'' you will find a lot more material.The authors have considered four different and all relevant application examples. The experimental section shows that the iFDM seems to work and that it can provide interesting results. The only comparison provided is against the FFBS-type algorithm, which we know will perform worse due to its construction. I know that it is a lot of work to implement other solutions to the problem, but if one were to do so it would probably provide an even better understanding of the performance of the model and it would be interesting to see the performance of existing solution to these problems. For example, for the multitarget tracking example, the simplest solution to this problem would probably be to use an extended Kalman filter together with nearest neighbour data association. Since your targets are very well separated I would expect this solution to perform quite well. It would be interesting to compare your performance against this simple standard solution. I have not worked with the cocktail party problem and the multiuser detection problems, but for the power disaggregation problem there are interesting solutions available, see for example the following NIPS paper (which is gaining some influence):Kolter, J. Z.; Batra, S.; and Ng, A. Y. Energy disaggregation via discriminative sparse coding. In Advances in Neural Information Processing Systems (NIPS). 2010. and the subsequent variational inference solution Kolter, J. Z., and Jaakkola, T. Approximate inference in additive factorial hmms with application to energy disaggregation. International Conference on Artificial Intelligence and Statistics. 2012.