Metric Learning for Temporal Sequence Alignment
In this paper, we propose to learn a Mahalanobis distance to perform alignment of multivariate time series. The learning examples for this task are time series for which the true alignment is known. We cast the alignment problem as a structured prediction task, and propose realistic losses between alignments for which the optimization is tractable. We provide experiments on real data in the audio-toaudio context, where we show that the learning of a similarity measure leads to improvements in the performance of the alignment task. We also propose to use this metric learning framework to perform feature selection and, from basic audio features, build a combination of these with better alignment performance.

The paper proposes an approach for learning a Mahalanobis metric for sequence alignment using a structured output prediction framework. It proposes to exploit a symmetrized version of the area loss between two alignments and shows how to make its optimization tractable. The paper is roughly well written but could be improved. Few forward references (e.g. using notations that have not been already defined) make a linear reading difficult. Also few claims that I list below are not so clear would maybe require more comments. -	Page 4 : I am not sure why if one signal is a reference \alpha would be steadily increasing, which is required for the area loss to be linear in Y.-	Page 5 : Second formulation of the symmetrized loss, defined on \bar{Y}(X): The fact that you work on this second formulation of the loss means that you optimize on \bar{Y}(x) rather than on Y(X) ? On page 6 the optimization is indeed performed on \bar{Y}. How to you get a solution in Y at the end? The experimental section show improved results when using your approach (using l_S) enabling the combination of features on the two datasets that you use. I have yet a few comments.On the first dataset you do not provide the results of an approach using uniform loss (i.e. using squared Euclidean distance instead of Mahalanobis), and it seems that you do not provide the best results from [14] either. Maybe because your experiments are not fully comparable with that of [14]? Also it seems you get much different results on this dataset as the ones published in [14] when looking at the accuracies reached with isolated features. You should probably add a discussion on these differences. Also why don’t you provide any comparative results of l_O?On the second dataset, you almost reach the performance of the best single handcrafted feature with the combination of your standard features. I understand it is already a good result but would your approach be able to reach even better results by combining all these handcrafted features?More generally I am not fully convinced by the experiments which miss alternative baselines. Also if one compares such alignment scores to what is learned in Hidden Markov models (HMMs) the learning of the Mahalanobis metric appears as a very narrow topic. Indeed learning a Mahalanobis distance in a Dynamic Time Warping alignment procedure would correspond to using a shared covariance matrix for all Gaussian distribution in a HMM, which is known to be a poor choice. From this viewpoint the topic of the paper, although it is relevant and the proposed method is smart and principled, appears of moderate importance.Minor remarks-	Bottom page 2 : Last term in bottom of the page one guesses what #Y_{t,s} is as well as what s is but it is not introduced.-	Caption of Figure 1. o	Red upper -> red lowero	Blue lower -> blue uppero	\delta_{abs} is undefined at this point.-	Caption of figure 2o	“the most violated constrained” -> “the most violated constraint”o	You say first that “we have represented the most violated constrained at the end of learning, when the training loss is the Hamming one or the symmetrized area loss” where these two cases are indeed much different on the figure. And then you say “Note also that, in terms of Hamming loss the most violated contraint for l_S and the Hamming one are the same, while they differ much for the area loss.” which seems contradictory with the first sentence. -	Equation 6 : missing ^2 for the first norm term -	Page 8 : missing part in the sentence : “Note also that, in terms of Hamming loss the most violated contraint for l_S and the Hamming one are the same, while they differ much for the area loss.”


Flexible Transfer Learning under Support and Model Shift
Transfer learning algorithms are used when one has sufficient training data for one supervised learning task (the source/training domain) but only very limited training data for a second task (the target/test domain) that is similar but not identical to the first. Previous work on transfer learning has focused on relatively restricted settings, where specific parts of the model are considered to be carried over between tasks. Recent work on covariate shift focuses on matching the marginal distributions on observations X across domains. Similarly, work on target/conditional shift focuses on matching marginal distributions on labels Y and adjusting conditional distributions P (X|Y ), such that P (X) can be matched across domains. However, covariate shift assumes that the support of test P (X) is contained in the support of training P (X), i.e., the training set is richer than the test set. Target/conditional shift makes a similar assumption for P (Y ). Moreover, not much work on transfer learning has considered the case when a few labels in the test domain are available. Also little work has been done when all marginal and conditional distributions are allowed to change while the changes are smooth. In this paper, we consider a general case where both the support and the model change across domains. We transform both X and Y by a location-scale shift to achieve transfer between tasks. Since we allow more flexible transformations, the proposed method yields better results on both synthetic data and real-world data.

This paper proposed a method to allow both the marginal and conditional distributions to change, which is less studied in the literature of transfer learning.In general, the paper is well written. But one thing is a bit vague for more general audience, i.e., why smoothness should be imposed and is so important in the proposed method. The authors should provide more details in the paper.This paper is motivated from [16] and [12], which assume the test X and Y can be transformed into new representations where the statistical property is somehow matched with corresponding domains. Though a bit incremental, the proposed method addressed an interesting transfer learning problem, and the experiments are sufficiently done to show the effectiveness of the proposed method.Minor issues:1. "mean-var-centered" should be "mean-var-scaled" in Figures 4 and 5.


Deep Networks with Internal Selective Attention through Feedback Connections
Traditional convolutional neural networks (CNN) are stationary and feedforward. They neither change their parameters during evaluation nor use feedback from higher to lower layers. Real brains, however, do. So does our Deep Attention Selective Network (dasNet) architecture. DasNet's feedback structure can dynamically alter its convolutional filter sensitivities during classification. It harnesses the power of sequential processing to improve classification performance, by allowing the network to iteratively focus its internal attention on some of its convolutional filters. Feedback is trained through direct policy search in a huge million-dimensional parameter space, through scalable natural evolution strategies (SNES). On the CIFAR-10 and CIFAR-100 datasets, dasNet outperforms the previous state-of-the-art model on unaugmented datasets.

This paper adds selective attention to a convolutional neural network and develops a method for training gating parameters for attention in order to optimize the network's classification performance. It addresses an important set of problems in computer vision -- how to adaptively reconfigure complex, deep processing stages when resources are limited and different tasks need to be solved.  The classification results are impressive, but the whole thing is very much a black box -- it suggests useful directions to pursue but does not reveal why the method works.  The analysis of image representation is not very revealing, and there is no sense of what the learned parameters (mean and variance of the theta distribution) are doing.  Nevertheless, the work is an important step beyond the current paradigm of large, feed-forward networks with parameters fixed after learning. The exposition is quite clear, though some of the equations have typos (or need to be explained better).major comments:I would like to see the authors seriously explore the connection to dynamical systems. The work seems closer to recurrent networks than to Reinforcement Learning.  The reward is observed immediately and the policy is deterministic, so training involves a traversal through the space of recurrent "policy" parameters theta to maximize the observed objective function.I did not find the analysis of the network processing the cat image very helpful in understanding how the new features of the model help recognition.   Why are layer 1 activations all increased at the first iteration?  Does this affect the objective function, or is the output invariant to global scalings?  Perhaps focusing on a smaller region of the image and analyzing the evolution of the gating variables in the region could be instructive.  Also, looking at difficult cases -- those misclassified by the standard Maxout network but correctly labeled by the proposed model, or teasing out the effect of the learned model parameters (mean and variance of theta) by constraining them further would be interesting.minor comments:Algorithm 1 description: where is h_M defined? is that the step of collecting observations?  F[i] and \Theta[i] are inside the loop over images (j), are the image fitness values collected in an array or overwritten on each iteration?Eqn 11: what is the sum over?  Is Eqn 8 for a single image, and should be indexed j?Text after eqn 11: what do 'x' and 'd' refer to?I am not clear on why regularization of theta is necessary, since theta is already sampled from the prior and will tend to small values anyway.  In fact, the last step of the optimization (gradient updates on the parameters after having sampled and evaluated the fitness of thetas) is not explained well (how do you compute the gradients) or justified theoretically (are you adjusting hyper-parameters of the posterior over policies).


Learning to Search in Branch-and-Bound Algorithms ⇤
Branch-and-bound is a widely used method in combinatorial optimization, including mixed integer programming, structured prediction and MAP inference. While most work has been focused on developing problem-specific techniques, little is known about how to systematically design the node searching strategy on a branch-and-bound tree. We address the key challenge of learning an adaptive node searching order for any class of problem solvable by branch-and-bound. Our strategies are learned by imitation learning. We apply our algorithm to linear programming based branch-and-bound for solving mixed integer programs (MIP). We compare our method with one of the fastest open-source solvers, SCIP; and a very efficient commercial solver, Gurobi. We demonstrate that our approach achieves better solutions faster on four MIP libraries.

The paper present a heuristic for node selection in Branch and Bound for Mixed Integer Programs based on machine learning.     Although machine learning used for node selection is not new the paper present a new approach (to the best of my knowledge).    They utilize a classifier together with an oracle for training two aspects: a node selection policy and a node pruning policy.    The first one is used to enforce a linear order/priority on the current open nodes of the Branch and Bound while the second one is used to further shrink the list of open nodes by pruningthe unpromising ones. The results look very promising compared to a state-of-the-art solver (Gurobi).The paper is well written and easy to follow, however I would have liked to read more details on their experimental setup. In particular it is not clear which kind of LP solver is used together with DAgger (is it Gurobi itself or another one?). Also some tuning parameters in page 6 are reported without explaining the process to arrive to such parameters.The related work section should also be strengthened. Here, I report a couple of references that might be worth considering:- "A Supervised Machine Learning Approach to Variable Branching in Branch-And-Bound" by Alvarez, Louveaux, Wehenkel- "Guiding Combinatorial Optimization with UCT" by Sabharwal, Samulowitz, ReddyDespite the encouraging results, a more thorough experimental section would be also helpful and insightful.For instance, it is not clear to me what is the main contribution that comes from the node selection independently from node pruning. A more extensive experimental result section showing these two aspects separetely would be insightful. Furthermore, despite the optimality gap compared to Gurobi looks indeed good, the method is inherently incomplete because of the pruning (as pointed out by the authors as well). How does it compare if only $\pi_s$ is used? A comparison based on a larger data set (not in term of instances but in term of different classes of problems) would also be welcome addition.Which were the features extracted for training that were the most relevant among the one you used?Finally, the presence of an oracle knowing the optimal solution is necessary in your setup. This is rarely the case when solving real-life problems. One could easily think of running an off-the-shelf solver with a reasonable timeout on those problems and then use the best solution found for training. The question would then be: how robust is the solution proposed when used with sub-optimal oracles?[Minor Comments]- I would suggest the author to use a grey-scale friendly version of figures (that otherwise are difficult to follow when printed in grey-scale)- Page 3, 3rd paragraph: there is a reference to a "Figure 3". Is it the correct reference?- Page 4, Figure 2: I believe the state machine describing the online algorithm is missing the case in which there are no feasible solutions.- Page 4, end of the page: "at any time there can be at most one optimal node on the queue". This is in general not true for a traditional Branch and Bound setting. You may want to specify that this applies only to the specific setup.Addendum: Thanks for answering a good portion of the questions above in the rebuttal.


On the Statistical Consistency of Plug-in Classifiers for Non-decomposable Performance Measures
We study consistency properties of algorithms for non-decomposable performance measures that cannot be expressed as a sum of losses on individual data points, such as the F-measure used in text retrieval and several other performance measures used in class imbalanced settings. While there has been much work on designing algorithms for such performance measures, there is limited understanding of the theoretical properties of these algorithms. Recently, Ye et al. (2012) showed consistency results for two algorithms that optimize the F-measure, but their results apply only to an idealized setting, where precise knowledge of the underlying probability distribution (in the form of the 'true' posterior class probability) is available to a learning algorithm. In this work, we consider plug-in algorithms that learn a classifier by applying an empirically determined threshold to a suitable 'estimate' of the class probability, and provide a general methodology to show consistency of these methods for any non-decomposable measure that can be expressed as a continuous function of true positive rate (TPR) and true negative rate (TNR), and for which the Bayes optimal classifier is the class probability function thresholded suitably. We use this template to derive consistency results for plug-in algorithms for the F-measure and for the geometric mean of TPR and precision; to our knowledge, these are the first such results for these measures. In addition, for continuous distributions, we show consistency of plug-in algorithms for any performance measure that is a continuous and monotonically increasing function of TPR and TNR. Experimental results confirm our theoretical findings.

Quality: Technical quality is very good, the theoretical results are sound and all the proofs are provided either in the main body of the paper, or in the appendix.Clarity: Despite a strong emphasis on theoretical results (numerous lemmas and theorems), the paper was actually relatively easy to follow and the quality of writing is very good. Nevertheless, I think the paper is very heavy on notation, and this should be improved. I give some suggestions below, but I leave it to the authors what changes (if any) they prefer to include. For instance, I would suggest to drop subscript D in the whole paper, i.e. TPR_D, TNR_D, P_D, regret_D, etc., would be TPR, TNR, P, regret, etc. It obscures the notational, while the distribution is always the same (D) and clear from the context. It will also be easy to distinguish expected quantities from empirical quantities, as the latter are always indicated by the "hat" symbol. Moreover, it is probably not needed to explicitly denote the dependence on S_1 and S_2 all the time (e.g., by using \widehat{\eta}_{S_1} or \widehat{t}_{S_2,\widehat{\eta}_{S_1},\Phi}). In the same way, \Psi does not need to be used in almost all symbols, as it is always clear what \Psi do we mean (there is only one \Psi analyzed throughout the whole proof). I understand the authors wanted to be precise with notation, but I think it is obscure at the moment and has a negative effect on the readability of the paper.Originality: As far as I am concerned, the results are novel and generalize some previous results shown for very specific cases (specific measure \Psi, stronger assumptions).Significance: Judging by interesting theoretical results and good experimental verification, I think the paper should have an impact on the theory of learning with non-decomposable performance measures.In my opinion, this is a good paper with interesting theoretical result. Although \Psi-consistency of L_1-consistent class probability estimator is not really that surprising, this is the first formal result of such generality for non-decomposable performance measures, and is worth publishing. In my opinion, the paper would benefit from answering to at least some of the questions below.- Is assumption that the optimal threshold exists and is in (0,1) necessary? If yes, it would be nice to give an illustrative counterexample, which shows why consistency fails when the optimal threshold is in {0,1}, or when it does not exist. If not, it would be worth trying to prove the theorems without this assumption.- Is Assumption A necessary (as compared to B) for non-monotonic performance measures? Again, I am missing an example of what can go wrong if \Psi is not monotonically increasing in TPR and TNR, while Assumption A is satisfied and Assumption B is not. If no such counterexample can be given, again, the authors should try to relax Assumption A.- It would also be interesting to consider the case when probability estimates and threshold are obtained on the same data set S (without splitting into S_1 and S_2). This is a common learning scenario in the real life, and hence it is legitimate to ask under what conditions consistency holds in such a setting?Some specific remarks:- Line 185-186: what does it mean "eta(x) conditioned on y=1 and on y=-1 is mixed"?- Line 192 (Theorem 1): "for some t^*_{D,\eta,\Psi} \in (0,1)" -- this symbol (t^*_{D,\eta,\Psi}) is already defined (what does the word "some" mean here), and it follows from Assumption A, that it is in (0,1), so this statement is unnecessary (I think).


Dimensionality Reduction with Subspace Structure Preservation
Modeling data as being sampled from a union of independent subspaces has been widely applied to a number of real world applications. However, dimensionality reduction approaches that theoretically preserve this independence assumption have not been well studied. Our key contribution is to show that 2K projection vectors are sufficient for the independence preservation of any K class data sampled from a union of independent subspaces. It is this non-trivial observation that we use for designing our dimensionality reduction technique. In this paper, we propose a novel dimensionality reduction algorithm that theoretically preserves this structure for a given dataset. We support our theoretical analysis with empirical results on both synthetic and real world data achieving state-of-the-art results compared to popular dimensionality reduction techniques.

The paper presents a dimensionality reduction method based for data that lives in a union of independent subspaces. The method is based on the observation that two independent subspaces collapse into two lines if a projection consisting of one vector from each subspace is applied on the data. The authors then present an algorithm to find vectors from different subspaces to be used for projection. Experimental results show that the proposed method performs better than state-of-the-art.In general, I believe that the paper presents a new approach, it explores a new idea, and it provides very good experimental results, and therefore eventually it's worth publishing. However, in my opinion, the paper is not written very well. The authors stress in multiple places that the idea  that 2K projection vectors are sufficient for K class data is "non-trivial". This might be the case, but it's a very intuitive result. I'm very surprised that the paper first presents a formal proof before the geometrical explanation, and this geometrical explanation is not very clear, and Figure 1 is very poorly designed. The proof is also very convoluted, for instance the same subscript j is used to both denote the components of vectors & matrices, as well as different subspaces. Section 3 can be re-organized into a theory and algorithm section, the current structure is not making this distinction clear.The authors can give simple directions how to go about proving Lemma 4 (why is this not Lemma 1?) In the same lemma, there's no guarantee that gamma_hat != 0. Is this a problem? Please comment. In line 179, I wouldn't use \Sigma for subspaces. Or if you do, it'll be good to briefly explain what addition of subspaces mean.It is clear that two major potential problem for the proposed method is noise, and cases where the independent subspaces assumption is not valid. I think the paper doesn't provide enough empirical evidence to investigate these cases. The study in Section 4.1 just verifies the theorems presented in the previous sections, which are already proven. What happens when the independence subspace assumption is not valid? It is not clear what value Section 4.1.2 and Figure 4 have. These can potentially be removed and a more-in-depth analysis can be added instead.The algorithms compared with in Section 4.2 are not state-of-the-art, not even close. There has been a large progress in this area (low rank and sparse subspace clustering), and I think it's unacceptable to ignore all of these recent methods and just compare with older methods like PCA and LDA..


Large Scale Canonical Correlation Analysis with Iterative Least Squares
Canonical Correlation Analysis (CCA) is a widely used statistical tool with both well established theory and favorable performance for a wide range of machine learning problems. However, computing CCA for huge datasets can be very slow since it involves implementing QR decomposition or singular value decomposition of huge matrices. In this paper we introduce L-CCA , a iterative algorithm which can compute CCA fast on huge sparse datasets. Theory on both the asymptotic convergence and finite time accuracy of L-CCA are established. The experiments also show that L-CCA outperform other fast CCA approximation schemes on two real datasets.

I acknowledge having read the authors' feedback. Given the clarification on the novelty aspects, I still feel that the technical contributions are minor even though the eventual algorithm is both useful and interesting. The authors claim Section 2.3 and Algortihm 1 to be novel, but several alternating algorithms for solving CCA via linear regression have been presented before (I did not mention these in the original review because I assumed Section 2 to be considered as background). Besides [20] and [22], for example Dehon, Filzmoser and Croux ("Robust methods for canonical correlation analysis", 2000), and Lykou and Whittaker ("Sparse CCA using a Lasso with positivity constraints", 2010) have in recent years presented CCA algorithms based on such derivation, citing earlier works from the 60s and 70s to motivate the re-formulation. I did not check whether all of the details are identical, but at least the relationship between the proposed algorithm and these earlier alternating LS derivations should be discussed in detail, so that readers would be able to see the novelty. If the details are the same, 2.3 cannot be considered novel.I now understand that the LING part has novel elements as well, namely the support for sparse matrices. However, this is not at all apparent in the writing and the derivations are very similar to the UAI paper. While solving CCA is indeed an excellent use-case for that technique, it is very straightforward and LING could also be considered as background. The resulting method might well be the best choice for learning large-scale CCA models, but this would be better presented for the community by a paper that does not hide the proofs in the appendix (but instead illustrates somehow what they mean in practice) and that has much more extensive experimentation, such as explicitly showing how slow the naive algorithm is, comparison of running times on sparse and non-sparse data, better study of the conditions where RP-CCA works and fails, and illustration on how the performance of the proposed algorithm depends on the the parameters k_{pc}, t_1 and t_2 to provide practical suggestions.(The rest is the original review)	The paper proposes an algorithm for efficient learning of CCA	for large sparse matrices, by iteratively solving learst	squares problems.  Different alternatives proposed by the	authors are compared on two  data sets with fairly good	results.	I am puzzled by the role of LING in this publication.  It is	mentioned in the background section with a citation to a very	recent arXiv publication, yet it is explained in Section 3 as	part of the new algorithm. I would like the authors to clarify	the exact relationship between these papers.  This is a	critical question since the eventual algorith is a direct	application of LING to Algorithm 1, which is not new in	itself, and the error analysis of Theorem 3 alone is not quite	sufficient to warrant publication in a top forum, especially	as it has been pushed to the Supplement.	Following the recent large-scale NLP applications of CCA, it	makes sense to consider fast alternatives. The motivation	seems to be in particular for solving setups with large p, for	which the starndard solutions are said to be quadratic or	cubic. There are, however, also solutions that are either	constant wrt p (kernel CCA solved in the dual space, for	example Hardoon et al. Neuroimage'07 uses KCCA with linear	kernels that are equivalent to CCA and p is only relevant when	computing the kernel) or linear in p (the more recent	probabilistic solutions; for example Klami et al. ICLR'14	presents a variant for sparse matrices).  It would make sense	to discuss also the relationship with such algorithms, to	clarify why they would not be as fast in practice as the	proposed method (KCCA is slow for large N, whereas	probabilistic CCA should suffer from the same problem as	RP-CCA, requiring large k to capture correlations with small	variance).	I like the experiments in the sense that they attempt to	illustrate how different variants work in different	conditions.  Comparison against RP-CCA is also very relevant,	since I believe for example [7] uses something like that, and	your results clearly show that RP-CCA is very good in certain	conditions (Fig. 2(a)) and poor in others. This is a valuable	result.  However, I am somewhat disappointed with the way the	experiments are conducted.  First of all, merely computing	correlations on the training data provides a very limited view	to CCA, especially as CCA on high-dimensional data overfits	ridiculously easily. Even though you get higher correlations	on training data compared to RP-CCA, the results could be	opposite on test data since focusing on dimensions with high	variance could reduce overlearning. This would imply that	the singular vectors are also off.  Hence, I would strongly	recommend reporting also correlations computed on leave-out	test samples.	Also, the test setup where you report results for three fixed	CPU time choices does not convey a very clear message.  Why	not just plot the sums of the first 20 correlations as a	function of (logarithmic) time, by running each algorithm on	multiple choices of the parameters?  Then show the actual	values like in Fig 1 and 2 for one snapshot. At least for me	this would tell more about how the algoritghms work.  It would	also be nice to somehow illustrate just how slow the naive	algorithms would be, perhaps by extrapolating the	computational time from a smaller data.	Quality: The method seems technically correct and the experiments	are satisfactory, albeit not optimal for illustrating the algorithms	in practice.	Clarity: The paper is easy enough to read, and the algorithm	boxes are very useful.	Originality: The method is rather straightforward and has	limited originality, since it simply plugs in LING for solving	CCA.  In the end, we are merely talking about a gradient-based	algorithm for solving a linear method that is known to be	solvable with least squares algorithms.	Significance: The paper has practical signficance since there	is clear demand for CCA implementations applicable on very	large data. This would further be improved by good open-source	implementation of the algorithm, good rules of thumb for	choosing the parameters, and extensive experiments.  The	theoretical significance is, however, quite limited. Maybe	some other forum would be more suitable for this kind of a	practical contribution.


Multi-Resolution Cascades for Multiclass Object Detection
An algorithm for learning fast multiclass object detection cascades is introduced. It produces multi-resolution (MRes) cascades, whose early stages are binary target vs. non-target detectors that eliminate false positives, late stages multiclass classifiers that finely discriminate target classes, and middle stages have intermediate numbers of classes, determined in a data-driven manner. This MRes structure is achieved with a new structurally biased boosting algorithm (SBBoost). SBBost extends previous multiclass boosting approaches, whose boosting mechanisms are shown to implement two complementary data-driven biases: 1) the standard bias towards examples difficult to classify, and 2) a bias towards difficult classes. It is shown that structural biases can be implemented by generalizing this class-based bias, so as to encourage the desired MRes structure. This is accomplished through a generalized definition of multiclass margin, which includes a set of bias parameters. SBBoost is a boosting algorithm for maximization of this margin. It can also be interpreted as standard multiclass boosting algorithm augmented with margin thresholds or a cost-sensitive boosting algorithm with costs defined by the bias parameters. A stage adaptive bias policy is then introduced to determine bias parameters in a data driven manner. This is shown to produce MRes cascades that have high detection rate and are computationally efficient. Experiments on multiclass object detection show improved performance over previous solutions.

This paper proposes a new cascading system for multiclass object detection. The algorithm exploits a new multiclass structurally biased boosting algorithm (SBBoost) that allows to introduce a bias towards some specific classes. SBBoost is used to implement a cascade where first models are biased towards the elimination of false positives (ie., the good detection of the negative class), late models are biased to finely discriminate the target classes, and middle models are biased in-between. The proposed MRes cascade is shown to outperform other cascades models on two object detection problems.The problem of multiclass object detection is important and has many applications. Building cascades for this setting is also very interesting to minimize detection times. Although several solutions already exist for this problem, the proposed method is original in that it solves the problem in a more integrated way than existing mutliclass cascading systems. A second contribution of the paper is the introduction of a new cost-sensitive multiclass boosting algorithm, called SBBoost. This algorithm seems also original but a comparison with related works on cost-sensitive multiclass boosting is missing in the paper (which focuses on the cascade aspect).The derivation of the SBBoosting method from the MCBoost method of Saberian and Vasconcelos is sound. It is first shown that MCBoost naturally introduces a bias towards difficult classes and this feature is exploited to introduce artificial class biases with the introduction of new bias parameters into the margin definition. Interestingly, these parameters can be interpreted as the logs of the cost of assigning examples from one class to another.The derivation of the MRes cascade model makes also much sense. I like the idea of going progressively from target detection to the discrimination of the classes. The proposed stage adaptive bias policy should clearly achieve this goal but it appears however to be not very well motivated from a theoretical point view. It is not clear which global criterion it optimizes and whether other policies could not achieve better performance.Experiments clearly show that the MRes cascades outperform other cascades on two different object detection problems. The compared methods are however rather complex and depends on several parameters that are not given in the paper. For this reason, while results of MRes are very good, we can only trust the authors that the other methods have been tuned appropriately for the comparison. With the information provided in the paper, it is clearly impossible to reproduce the experiments.I’m also disappointed that the proposed method is only evaluated as a whole. There is no analysis of the effect of its parameters or evaluation of the different design choices. For example, it would be interesting to turn off the stage adaptive bias policy to see how important it is to achieve good performance. What is the performance achieved by the MCE method without adaption? The newly proposed SBBoost method is also not validated per se. It would be interesting to compare it with other cost-sensitive multiclass boosting methods. Is this method crucial in the context of the cascade or could it be replaced by another cost-sensitive approach with similar performance?The paper is well written. It is mostly clear and well organized. While all individual steps of the algorithm are clear, it is not so clear however how all pieces are put together. It should help to have a pseudo-code description of the method. For example, while I think I understand how the cascade is trained, I’m not sure how exactly it is used to compute predictions. In Equation (1), what are the h_i(x) models? Is h_i(x) the ith weak learner or does it correspond to the sum of the predictions from all weak learners until the ith one? I’m not sure also to understand how the update of the beta is interleaved with the adaptive policy of (22).Minor comments:- The conclusion is not very useful as it is a simple summary of the paper without any discussion of the method or of future works.- Page 5,  « \beta is increased to (\beta+1)/2 » -> decreased


Deep Symmetry Networks
The chief difficulty in object recognition is that objects' classes are obscured by a large number of extraneous sources of variability, such as pose and part deformation. These sources of variation can be represented by symmetry groups, sets of composable transformations that preserve object identity. Convolutional neural networks (convnets) achieve a degree of translational invariance by computing feature maps over the translation group, but cannot handle other groups. As a result, these groups' effects have to be approximated by small translations, which often requires augmenting datasets and leads to high sample complexity. In this paper, we introduce deep symmetry networks (symnets), a generalization of convnets that forms feature maps over arbitrary symmetry groups. Symnets use kernel-based interpolation to tractably tie parameters and pool over symmetry spaces of any dimension. Like convnets, they are trained with backpropagation. The composition of feature transformations through the layers of a symnet provides a new approach to deep learning. Experiments on NORB and MNIST-rot show that symnets over the affine group greatly reduce sample complexity relative to convnets by better capturing the symmetries in the data.

This paper proposes to extend usual convolutional networks from thetranslation symmetry group to the affine group (symnets). This entailscomputation that may grow as k^6 instead of k^2 when each feature iscomputed on a window of width k in each of the group dimensions.  To dealwith this computational complexity, the paper proposes to optimize thelocation (in R^6) of N control points located in the space of the group,for each feature map.  The output of the convolution+subsampling can thenbe approximated by a form of weighted averaging between the control points,and the whole thing can still be optimized by backprop + gradient-basedoptimization.This is a really interesting idea and it is also new (the part about usingoptimized control points, not the part about extending convnets totransformations other than translation) as well as attacking an age-oldproblem, but the weak point of the paper are the experiments, along withthe selling tone used in many places. The results suggest on the surface that symnets can generalize better onvery small datasets (a few hundred examples), but the appropriate controlexperiments are not performed, and both the conclusions (and generally thelanguage of the paper) seem not in line with the standards of scientificreporting, in an attempt to make the proposal and the results look better.The experiments should have compared with a convnet that uses the sameprior information, i.e., with an augmented dataset, randomly transformingalong the affine group (which is the standard practice anyways). Theexperiments should have compared networks of arbitrary depth and size(optimized on the validation set), and not be restricted to networks of thesame size and depth, because it is likely that in order to emulate a 6-Dsymnet, a regular convnet needs more resources (both in depth and number offeatures and parameters), but these can be compensated by using anaugmented training set. Finally, a fair comparison should includecomputation time (especially for training). One might suspect that theproposed algorithm (which is not 100% clear, although the big picture isclear) requires a lot more computation per training example than a regularconvnet. The reader needs to know how much more, in the specificexperiments reported. A number of elements of that paper are not acceptable in the scientificreporting standards: hiding the above computational cost information (ofwhich surely the authors are aware - not a single word is spoken abouttraining and test time), not doing the above proper and fair comparisons,and then claiming that symnets generalize better without stating that theadvantage is mostly for tiny datasets with a few hundred examples. It isunfortunate that many authors have a similar behavior, but still, it shouldnot be acceptable at NIPS.Additional detailed pointsLine 39: The authors write that convnets require enormous datasets. But thereare such datasets, so the question is whether one can do better with the givendatasets. It is true that we would like to generalize with few examples fornew classes for which there are not such large number of labeled examples.The use of the word 'kernel' initially threw me off. Of course it is used herein a sense different from the more common use in the ML community. I suggestto clarify that early on.Line 69: "we do not completely pool out a symmetry", but neither do convnets.Similarly for discriminative training, same line. Clarify that you refer tosome other approach, maybe.Line 241: "polynomial dependence on D", actually should be exponential in D,and polynomial in k, i.e. simply k^D.Line 290: why use the minimization of squared error rather than maximizationof the dot product? Is the norm of the feature vectors kept constant? how? Line 303: "penalizing relative locations [10]". Not having read [10], the readerwould like to know briefly what this is about.Figures 5, 6: it is crazy to compare only with a single-layer or a 2-layer convnet.See main discussion above.Section 6.2: what was N, at the end of the day?Line 427: I found this conclusion really misleading "affine symnets achievedhigher accuracy than convnets while using significantly less data". The truthis that when *only* a few hundred examples are available, and when the convnetcannot exploit transformed augmented datasets and is highly limited in capacity(all of which are not realistic), then the symnets generalize better. Clearlythis is a very different kind of message...Line 431: what about the other hyper-parameters?=========== POST-REBUTTAL NOTESThe authors have argued that "data augmentation is not feasible for many datasets" to justify the fact that they did not want to compare symmetry nets with convnets + data augmentation. The truth is that data augmentation has been for many years the standard way of training convnets in practice, it is cheap, simple, and effective, and helps to embody the same kind of prior information that symmetry nets try to exploit. Data augmentation does not need to visit the cross-product of all possible deformations: that's the beauty of it. A random sample of deformations along all the desired directions of invariance is sufficient to give a huge boost to convnets ESPECIALLY ON SMALL DATASETS like the ones the authors used.  If symmnets can exploit that same prior information better than random deformations, then this is a very important piece of information! The authors have also refused to change their misleading conclusion. That is sad. Most careful readers will not be fooled but will instead come out with a bad impression about the authors.Nonetheless, I have raised my rating above the acceptance threshold because I believe I like the underlying idea, which is original and should be a good contribution to NIPS.I strongly encourage the authors to reconsider the above two points in order to make their paper a more credible scientific contribution that will be more likely to have impact in the field.


Communication-Efficient Distributed Dual Coordinate Ascent
Communication remains the most significant bottleneck in the performance of distributed optimization algorithms for large-scale machine learning. In this paper, we propose a communication-efficient framework, COCOA, that uses local computation in a primal-dual setting to dramatically reduce the amount of necessary communication. We provide a strong convergence rate analysis for this class of algorithms, as well as experiments on real-world distributed datasets with implementations in Spark. In our experiments, we find that as compared to stateof-the-art mini-batch versions of SGD and SDCA algorithms, COCOA converges to the same .001-accurate solution quality on average 25× as quickly.

The paper describes a parallel variant of SDCA algorithm which each machine executes SDCA updates using local parameters and these local parameters are periodically averaged.  Theorem 3 provides the convergence guarantee of the algorithm.Quality: Unfortunately, Theorem 3 is not very interesting since the bound is a function of T/K; as T should be a linear function of K at best, there will be no speed-up by increasing K, the number of workers.  What this means is that if we use beta_K = 1, which is the only case the theorem covers, then the algorithm will become more and more conservative as we increase the number of machines, and compared to the case of single-thread execution the algorithm will not be any faster.  In practice we could make the algorithm more aggressive by using beta_K > 1, but the paper does not provide theoretical support.  The empirical comparison against other methods seems motivating, but unfortunately an experiment on scaling behavior of the algorithm as a function of K is missing, which I believe is critically important considering Theorem 3 does not guarantee speed-up.Clarity: Lemmas and theorems in the paper are well-organized and easy to follow, since authors have done a good job in providing intuitive explanation of the proof.Originality: The proposed algorithm is a generalization of a popular heuristic which local parameters are periodically averaged, and the proof does seem to provide new techniques or insights; the proof of Lemma 2 does not depart much from the original proof for SDCA, and then the main theorem (Theorem 3) is just an application of Jensen's on top of Lemma 2.  This is why Theorem 3 is weak and does not provide any speed-up guarantee;  other parallel algorithms such as 11 or 21 departs from the proof of single-thread case by taking account of correlation between data points and therefore are able to provide speed-up guarantees.  I believe it should be necessary as well for parallel SDCA algorithm to consider correlation between data points, if one wants to achieve a nontrivial convergence guarantee.Significance: The paper provides a nice insight that H, the period of synchronization, might have to be a function of C, the local convergence speed.  However, this insight is based on a loose convergence guarantee (Theorem 3) which does not provide any parallelization speedup.  Even worse, since C is not known a priori, one cannot choose H without actually executing the algorithm.


Simple MAP Inference via Low-Rank Relaxations
We focus on the problem of maximum a posteriori (MAP) inference in Markov random fields with binary variables and pairwise interactions. For this common subclass of inference tasks, we consider low-rank relaxations that interpolate between the discrete problem and its full-rank semidefinite relaxation. We develop new theoretical bounds studying the effect of rank, showing that as the rank grows, the relaxed objective increases but saturates, and that the fraction in objective value retained by the rounded discrete solution decreases. In practice, we show two algorithms for optimizing the low-rank objectives which are simple to implement, enjoy ties to the underlying theory, and outperform existing approaches on benchmark MAP inference tasks.

This paper studies the technique of "low-rank relaxations" for MAP inference.The idea is to solve the problem $max_{x\in +-1} x^T A x$ using a relaxed form $max_{X\in \R^{n\times k}} tr(X^T A X)$ where the second maximization is constrained so that rows of $X$ have unit length. The value of $k$ can be varied to get different kinds of approximation. Solutions to the second problem are turned into solutions to the first using "randomized rounding" in which the rows of $X$ are projected onto a random vector and rounded to plus or minus 1.This idea is not new but the present paper contributes theoretical bounds which guide the choice of $k$. The mathematics was interesting to me and I did not find any mistakes, although I did not check the proofs of any of the theorems. The paper's conclusion is that $k$ does not have to be very large to get good results.The relaxation-rounding ("R^3") technique was compared to Mean Field and Gibbs on synthetic data with good results. I especially liked the argument in Figure 2 to show that R^3 can avoid some local optima which trap Gibbs and MF.I also liked the emphasis on the flexible interface provided by the algorithm, allowing different treatment of sparse and dense weight matrices for example (Introduction, penultimate paragraph).I was not clear on how novel the R^3 algorithm is, apparently from section 6 paragraph 3, similar algorithms have been presented before, but with different emphasis and no theory. It would be good to be able to convey this earlier in the paper, and tell the reader explicitly if "R^3" is a new term that is being coined here...End of 4.3: "every variables"There is an argument in the appendix that we need only consider MRFs with pairwise interactions since an auxiliary variable can be introduced to simulate singleton factors. If the auxiliary variable has the wrong value in a solution, simply flip the sign of all variables, and the objective will be preserved due to sign-symmetry of the pairwise-only MRF. It would be nice to see this argument appear early in the paper since otherwise some readers won't realize that the algorithm is fully general.Quality: I think the paper is of high quality.Clarity: It is clear and I enjoyed reading it and learning some new mathematics.Originality: The paper is not terribly original, but seems to provide some important results on an algorithm of interest.Significance: I think it is reasonably significant. MAP is not my favorite problem, but it's useful in some areas, and studying it gives rise to interesting mathematics which may have wider applicability. I think the paper will be interesting to Bayesians as well as MAP people.


Automatic Discovery of Cognitive Skills to Improve the Prediction of Student Learning
To master a discipline such as algebra or physics, students must acquire a set of cognitive skills. Traditionally, educators and domain experts use intuition to determine what these skills are and then select practice exercises to hone a particular skill. We propose a technique that uses student performance data to automatically discover the skills needed in a discipline. The technique assigns a latent skill to each exercise such that a student's expected accuracy on a sequence of same-skill exercises improves monotonically with practice. Rather than discarding the skills identified by experts, our technique incorporates a nonparametric prior over the exerciseskill assignments that is based on the expert-provided skills and a weighted Chinese restaurant process. We test our technique on datasets from five different intelligent tutoring systems designed for students ranging in age from middle school through college. We obtain two surprising results. First, in three of the five datasets, the skills inferred by our technique support significantly improved predictions of student performance over the expertprovided skills. Second, the expert-provided skills have little value: our technique predicts student performance nearly as well when it ignores the domain expertise as when it attempts to leverage it. We discuss explanations for these surprising results and also the relationship of our skilldiscovery technique to alternative approaches.

This paper proposes a probabilistic approach for learning the assignment of exercises to skills from student data, where student knowledge changes while exercises are being solved; the model also estimates the student knowledge while estimating the skill assignments. The paper uses a weighted CRP to model the assignment, incorporating expert labelings through the weighting. In simulation, the method recovers skill labelings with high accuracy, with little dependence on the expert labels, and across several datasets, the paper finds that skill labelings from this method result in higher prediction accuracy than other approaches. Overall, I found the paper to be clear and the proposed model is a relatively novel extension of existing methods. The evaluation methods demonstrate that the approach is likely to be effective for relevant applications, and that its inferences are accurate. I appreciated the inclusion of both synthetic data and a variety of real datasets. One interesting issue I found in the evaluation is that while the single skill assumption seems partially justified by the fact that these exercises were created to tap a single skill, the expert skills are in many cases a poor fit for prediction, suggesting that the experts may not be clear about the connection between each item and skill. Some discussion of the poor fit of the expert model is included in the discussion, but I think this remains an important point to consider, both in the motivation of the model and in trying to determine what effect using these new skills might have for control policies for tutors. One point that is not touched on is whether ordering/selection effects in terms of what items each student performs and in what order may skew the results; it would be great if a small amount of space in the paper could be devoted to addressing this issue, given that in typical datasets from tutoring systems one cannot count on items being selected randomly or administered in random order. The main concern I had with the paper was the presentation of alternative methods, specifically in the paragraphs around lines 57-94 of the intro. The paper first claims that automated techniques either "rely on an expert labeling which they then refine or treat the student knowledge as static." However, a later paragraph notes the Gonzalez-Brenes and Mostow work which does not make this assumption (note: I am assuming this is what is meant, rather than the listed reference numbers of 8-9 which are BKT). Additionally, the matrix factorization methods that are mentioned seem to be cases of "treating the student knowledge as static" and thus might be cited in this sentence as well (e.g., Desmarais' work or the SPARFA work). I noticed that the Lan et al. factorization work was included in the references, but could not find it in the main paper; it also seems relevant here and is an example of a model with time-varying student knowledge that does not rely on an expert labeling.Update: Thanks for the clarification in the rebuttal about the references - I'm leaving the notes here in case it's helpful to make sure things are fixed and because I think the apparent contradiction between the claims about existing methods and the Gonzalez-Brenes and Mostow work should be addressed, but the rebuttal addresses my concerns.  Also in the introduction, the paper notes that the "best models produced by matrix factorization have had low-dimensional skill spaces, specifically N_skill = 2"; it would be helpful to provide a citation here to clarify the context, given that this is not true for a number of results reported in papers about creating Q-matrices (e.g., Tatsuoka's original fraction work, Desmarais's refinement of these Q-matrices).  Minor note:On line 425, the same issue with reference numbers occurs as in the intro (8-9 are substituted when I suspect 10-11 are intended).


Unsupervised Deep Haar Scattering on Graphs
The classification of high-dimensional data defined on graphs is particularly difficult when the graph geometry is unknown. We introduce a Haar scattering transform on graphs, which computes invariant signal descriptors. It is implemented with a deep cascade of additions, subtractions and absolute values, which iteratively compute orthogonal Haar wavelet transforms. Multiscale neighborhoods of unknown graphs are estimated by minimizing an average total variation, with a pair matching algorithm of polynomial complexity. Supervised classification with dimension reduction is tested on data bases of scrambled images, and for signals sampled on unknown irregular grids on a sphere.

This paper uses transformations of the form  |W|\pi to build deep networks, where W is a discrete Haar transform, | | is the pointwise absolute value nonlinearity, and \pi is a permutation of the index set of the input vector.    The authors note that the \pi at each layer can be optimized for additive costs of pairs of indices in O(d^3) operations where d is the size of the input.  They  propose  a cost function that maximizes the variance of the output of a layer.  On the MNIST data set and a artificially transformed MNIST problem they use their (unsupervised) feature maps as inputs to an SVM and get strong classification results.This is an interesting paper and should be accepted.  The approach is unusual: they manage to arrange for a tractable combinatorial optimization for feature learning; after the optimization, the fprop is very fast.  I can't think of another work that does this.    On the other hand, the approach is not very well motivated.  The authors use a abs nonlinearity, why?  in their scattering network references, abs nonlinearities are used to introduce invariance to deformations; the choice of abs is well motivated because of its interactions with specially designed complex filters.   Why do we expect an abs nonlinearity to reduce nuisance variation more than variation that we care about generically/unsupervised, and with real haar type filters?  Moreover, the authors justify the cost function they use as the one that limits the contractiveness of their representation.  What stops the reductions in contractiveness from being precisely reintroduction of variance in nuisance dimensions?  The experiments do not give much intuition here: the MNIST data set is very special (it has well defined clusters), and previous works using sparse coding have gotten equally strong (permutation invariant) results (e.g. in the works of Julien Mairal and coauthors; although there the feature learning was supervised).    I would have loved a page or two less about filter banks etc. and more experiments on other data sets, or more analysis of what is happening here.minor details:  citation 11 the authors names are mangled, on line 212 \pi^0 appears twice, 3.1 needs to have a more explicit/clearer  definition of interlacingAll in all, interesting enough that it should really be accepted, but experiments are lacking: a strong result on MNIST is not especially illuminating or noteworthy.


Efficient Optimization for Average Precision SVM
The accuracy of information retrieval systems is often measured using average precision (AP). Given a set of positive (relevant) and negative (non-relevant) samples, the parameters of a retrieval system can be estimated using the AP-SVM framework, which minimizes a regularized convex upper bound on the empirical AP loss. However, the high computational complexity of loss-augmented inference, which is required for learning an AP-SVM, prohibits its use with large training datasets. To alleviate this deficiency, we propose three complementary approaches. The first approach guarantees an asymptotic decrease in the computational complexity of loss-augmented inference by exploiting the problem structure. The second approach takes advantage of the fact that we do not require a full ranking during loss-augmented inference. This helps us to avoid the expensive step of sorting the negative samples according to their individual scores. The third approach approximates the AP loss over all samples by the AP loss over difficult samples (for example, those that are incorrectly classified by a binary SVM), while ensuring the correct classification of the remaining samples. Using the PAS-CAL VOC action classification and object detection datasets, we show that our approaches provide significant speed-ups during training without degrading the test accuracy of AP-SVM.

This paper proposes three different approaches to speed up the learning for AP-SVM (a structured SVM with AP loss).1. The first approach uses the unimodal behavior of the loss function, so uses binary search to speed up calculating the loss.2. The second approach notices that usually we have more negative samples compared to positive ones, and in calculating the loss, relative rank of negative samples falling between two consequent positive ones is not important. Hence, it speeds up by identifying all negatives between two consequent positive ones and assigns the same rank to all of them.3. finally, it approximates the training by using 0-1 loss for easy examples (easy under 0-1 loss SVM).In my view, all three tricks are interesting, clever, and effective in speeding up the training. They are described clearly. In working with big data, speeding up the training is an important problem, so NIPS audience are interested in this topic.I think the evaluation can be more thorough. Particularly, it might be important to try the algorithm on a different problem and also with larger number of training data. For instance, as I understood, the number of negatives in this task is not that large. There are other tasks like object detection that any possible bounding box in the image is considered a negative sample, so the number of negative samples can be exponentially large. I would suggest applying this algorithm on one of those tasks to show off the real power of this algorithm.In Table 1, I was thinking AP-SVM-SELECT should be faster than AP-SVM-SEARCH, but it is not the case. It might be the case in another experiment with larger number of examples. Also, in Fig 2-right, I was thinking by increasing the number of positive examples, AP-SVM-SELECT should loose its advantage and become slower than AP-SVM-SEARCH since the gaps become small, but it is not the case in this Fig.In Fig 2 middle, I am surprised that the binary SVM is slower than the AP-SVM-SEARCH. I thought the only difference is in calculating the loss, and calculating 0-1 loss is trivial. Maybe I didn't understand it completely.I think these should be discussed and explained if possible in the text.Is it possible to combine the first two tricks? Or even combine all three? What do we expect to see? Line 21 in abstract says they are complementary, so should be nice to show it.Minor points:Line 431, "hard" should change to "easy" to be consistent with Line 344. Am I right?Line 334, in "is less than 1", I think "1" should change to "-1".


Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)
We present the first provably sublinear time hashing algorithm for approximate Maximum Inner Product Search (MIPS). Searching with (un-normalized) inner product as the underlying similarity measure is a known difficult problem and finding hashing schemes for MIPS was considered hard. While the existing Locality Sensitive Hashing (LSH) framework is insufficient for solving MIPS, in this paper we extend the LSH framework to allow asymmetric hashing schemes. Our proposal is based on a key observation that the problem of finding maximum inner products, after independent asymmetric transformations, can be converted into the problem of approximate near neighbor search in classical settings. This key observation makes efficient sublinear hashing scheme for MIPS possible. Under the extended asymmetric LSH (ALSH) framework, this paper provides an example of explicit construction of provably fast hashing scheme for MIPS. Our proposed algorithm is simple and easy to implement. The proposed hashing scheme leads to significant computational savings over the two popular conventional LSH schemes: (i) Sign Random Projection (SRP) and (ii) hashing based on p-stable distributions for L 2 norm (L2LSH), in the collaborative filtering task of item recommendations on Netflix and Movielens (10M) datasets.

This paper generalizes the LSH method to account for the (bounded) lengths of the data base vectors, so that the LSH tricks for fast approximate nearest neighbor search can exploit the well-known relation between Euclidian distance and dot product similarity (e.g. as in equation 2) and support MIPS search as well.  They give 3 motivating examples where solving MIPS vs kNN per se is more appropriate and needed.  Their algorithm is essentially equation 9 (using equation 7 compute vector reformulations Q(q) and P(x) of the query a database element respectively). This is based on apparently novel observation (equation 8) that the distance from the query converges to the dot product plus a constant, when a parameter m which exponentiated the P(x) vector elements is sufficiently large (e.g. just 3 is claimed to suffice, leading to vectors Q(q) and P(x) which are just that m times larger than the original input dimensionality.The work seems novel and potentially significant.  One concern though is that the experimental results actually provided do not clearly demonstrate an example where this method is better in practice than brute force linear scan.  The authors claim their "scheme thus offers around  5x speedup over linear scan", at the end of Section 4.2, but they do not clearly state how they implemented linear scan or how much wall clock time linear scan took.  The reason this is a concern is that linear scan can be implemented on modern hardware (multicore CPUs or GPUs) to be ~10x faster using modern libraries (e.g. BLAS SGEMM matrix multiply operatons, using batches of queries and the fully batch of database vectors) than the vector based, one-query-at-a-time loops that presumably the proposed hased-baed scheme requires, due to the cache-awareness of tile-based SGEMM ops for modern hardware.  Furthermore, hashing schemes will have additional overheads that must be demonstrated to not add significance constants -- especially when the speedup versus simple, low-constant-overhead linear scan approaches are relatively low (such as 5x here).  In short, this paper's focus on the 5x less dot products being performed for 90% accuracy does not prove that the method actually is, or will be for practical problems in general, faster in a practical wall clock sense than well-implemented linear scans.


Structure learning of antiferromagnetic Ising models
In this paper we investigate the computational complexity of learning the graph structure underlying a discrete undirected graphical model from i. Aside from structural assumptions on the graph such as it being a tree, hypertree, tree-like, etc., many recent papers on structure learning assume that the model has the correlation decay property. Indeed, focusing on ferromagnetic Ising models, Bento and Montanari [3] showed that all known low-complexity algorithms fail to learn simple graphs when the interaction strength exceeds a number related to the correlation decay threshold. Our second set of results gives a class of repelling (antiferromagnetic) models that have the opposite behavior: very strong interaction allows e cient learning in time Â O(p 2 ). We provide an algorithm whose performance interpolates between Â O(p 2 ) and Â O(p d+2 ) depending on the strength of the repulsion.
Aside from structural assumptions on the graph such as it being a tree, hypertree, tree-like, etc., many recent papers on structure learning assume that the model has the correlation decay property. Indeed, focusing on ferromagnetic Ising models, Bento and Montanari [3] showed that all known low-complexity algorithms fail to learn simple graphs when the interaction strength exceeds a number related to the correlation decay threshold. Our second set of results gives a class of repelling (antiferromagnetic) models that have the opposite behavior: very strong interaction allows e cient learning in time Â O(p 2 ). We provide an algorithm whose performance interpolates between Â O(p 2 ) and Â O(p d+2 ) depending on the strength of the repulsion.

This paper presents some theoretical results for the complexity of learning pairwise graphical models. Arguably, the most important result is a lower bound of the complexity of the general case. I believe the initial part of the paper can be improved. For instance, a "graphical model" is never defined and the discussion of sect.1.1 and 1.2 are confusing because of that. While I understand what the authors want to say, they should formally define the problem of interest, including the graphical model and the objective function that they want to optimise as being the problem of "learning the structure of general graphical models". For instance (and if I understood correctly), the work considers graphical models with pairwise relations only, which could be emphasised in the initial description of the problem (currently missing, even though the paper later clarify most of my doubts, reading its initial part for the first time was not much informative). The goal is explained as "recovering a graph on p nodes", but this is too informal. The problem that is being solved only becomes clear in sect.2, and even there it is not really formally defined. Because of the presentation, theorem 1.1 is not so appealing in the first time one reads it. In a different note, the initial complexity discussion ignores amount of data to learn, although this is clarified later. All in all, the paper seems sound and with an interesting theoretical study. However, I admit not to be an expert on this exact topic, and it is not easy to me to assess well the relevance and novelty. For example, the main result (of theorem 1.1) does not seem surprising to me, but if it had not been proved before, its proof is of clear relevance.Minor issues:- "Thus algorithms... does" -> do- "turns out reconstruct" ?- Algorithm 1: S should start empty, algorithm 2 and 3, hat{E}, etc. Better at least to make clear that sets are assumed to start empty.- "Thus the ML estimate contains a subset of those edges e which have not..." Could you explain this with slightly more detail?


Robust Logistic Regression and Classification
We consider logistic regression with arbitrary outliers in the covariate matrix. We propose a new robust logistic regression algorithm, called RoLR, that estimates the parameter through a simple linear programming procedure. We prove that RoLR is robust to a constant fraction of adversarial outliers. To the best of our knowledge, this is the first result on estimating logistic regression model when the covariate matrix is corrupted with any performance guarantees. Besides regression, we apply RoLR to solving binary classification problems where a fraction of training samples are corrupted.

The manuscript proposes a new method for robust logistic regression with a focus on dealing with outliers with high leverage. Outliers are assumed to come from an arbitrary and unknown distribution, but the number of outliers is assumed to be known ahead of time, which is a theoretically convenient, but practically slightly more troubling assumption.The method is based on maximising the sum of y*x'beta, but only summed over the observations which contribute in absolute terms the least to the objective function, thus outliers with large leverage are excluded. This can be elegantly translated into a linear programming problem.The authors derive some risk bounds. In common with most of these bounds, they are probably too loose to have a practical quantitative use, but are useful for qualitative interpretation.The manuscript contains a small simulation study in which the proposed method compares very favourably with classical logistic regression.A few more detailed comments:- Why do you need the preprocessing step? Observations with large ||x|| should also be among the observations with large y*x'beta and thus omitted by the algorithm. Also T is decreasing in n (and tends to 0 in the limit), so the more data we have, the more observations are thrown out by the preprocessing. For example, what proportion of observations is removed by the preprocessing step in the simulation?Also, I am surprised that T is not chosen based on n or lambda. - On a related point, it might be worth looking at how well logistic regression does after the same preprocessing is carried out.- Given the preprocessing and the set-up of the method, situations where (like in the simulation shown) the outliers have much larger variance in the covariates are quite favourable to the proposed method (they are very likely to trigger the preprocessing criterion or lead to a large contribution to the objective function and are thus omitted). What happens is sigma_o is decreased, so that outliers look less like obvious outliers and are more likely to sneak into the n smallest contributions (and thus not excluded by the method)? Will the method perform better or worse?- One important question is how much worse the method performs when compared to logistic regression when there are no outliers. The very left end of figure 2 suggests no big difference in terms of estimating beta, but a more substantial difference when looking at the misclassification rate. Essentially, what price do we have to pay for robustness?- In practice, how do I choose n? How sensitive is the method to the choice of n?The manuscript is well written and clearly structured.


Controlling privacy in recommender systems
Recommender systems involve an inherent trade-off between accuracy of recommendations and the extent to which users are willing to release information about their preferences. In this paper, we explore a two-tiered notion of privacy where there is a small set of "public" users who are willing to share their preferences openly, and a large set of "private" users who require privacy guarantees. We show theoretically and demonstrate empirically that a moderate number of public users with no access to private user information already suffices for reasonable accuracy. Moreover, we introduce a new privacy concept for gleaning relational information from private users while maintaining a first order deniability. We demonstrate gains from controlled access to private user preferences.

This paper proposes a privacy-preserving mechanism for recommender systemswhich are based on the popular matrix factorization. Specifically, theauthors propose splitting users into disjoint groups of public and privateusers. Private users do not share any information (ratings) with thesystem while public users share all of their information. The authors showthat under certain technical conditions one can bound the estimationaccuracy of the item features based on the number of observed ratings.This estimate can further be used to bound the reconstruction error of theprivate users. The authors further demonstrate the empirical performanceof their approach using the Movielens 10M dataset.Privacy in recommender systems is an important problem that we are onlybeginning to explore. As far as I know most formal previous approachesrely on differential privacy. This paper considers that even therecommender engine is not safe. This seems like a reasonable practicalsetting. In that setting the authors develop an interesting framework.The formalism developed in this paper is clear and seems sound as are thederivations. The results (bounding the error of the item factors andconsequently the errors of the reconstruction) are interesting. It wouldbe nice to give a bit more intuition about Theorem 3.5.Overall the paper is well written and easy to follow.The experiments are also reasonable. They demonstrate what is needed andthe comparisons to other methods is reasonably convincing.In the first set of experiments, it would be good to clarify what thelabel "Percentage of Users" mean. I understood it to mean the percentageof all users that were public.The second set of experiments uses 100 public users and up to 400 privateusers. It is a bit unclear why the authors are not reporting results withmore private uses (the dataset contains 10K users). What happens in thatsetting? Do any of the DP methods (especially LAP eps=5) reach similarperformance as PMC and PMD. Was there a reason for stopping at 400 privateusers? It would be good to show that PMC and PMD do well in a lesssynthetic setting. It would also be nice to provide results in the casewhere the private users aren't the ones that have necessarily consumed themost items.


Covariance shrinkage for autocorrelated data
The accurate estimation of covariance matrices is essential for many signal processing and machine learning algorithms. In high dimensional settings the sample covariance is known to perform poorly, hence regularization strategies such as analytic shrinkage of Ledoit/Wolf are applied. In the standard setting, i.i.d. data is assumed, however, in practice, time series typically exhibit strong autocorrelation structure, which introduces a pronounced estimation bias. Recent work by Sancetta has extended the shrinkage framework beyond i.i.d. data. We contribute in this work by showing that the Sancetta estimator, while being consistent in the high-dimensional limit, suffers from a high bias in finite sample sizes. We propose an alternative estimator, which is (1) unbiased, (2) less sensitive to hyperparameter choice and (3) yields superior performance in simulations on toy data and on a real world data set from an EEG-based Brain-Computer-Interfacing experiment.

840:this is a very nice paper, with compelling theoretical, simulated, and real data results.  i have a few majorish issues, and some minor ones.major-- one can choose lambda via CV or some theoretical tool.  if the theoretical tool has no parameters, it is a clear win.  however, there is a truncation parameter here. this manuscript did not convey to me *how* to choose b, and importantly, the extent to which the results are robust to this choice of b.  if this method is to be adopted as the de facto standard, some discussion about how to choose b and robustness to that choice is necessary.-- given that the main justification of using this method over CV is computational time, one might also acknowledge that practitioners always weigh a trade-off between accuracy and time.  clearly, this method is faster than CV, assuming we have a good way of choosing b.  but, how accurate is it? if it is much less accurate, than the improvement in time might not be so useful.  for example, in the real data example, we could simply use the average class covariances for the other subjects.  this would be fast, parameter free, and maybe just as accurate?minor-- in eq 5, b is some constant that satisfies some properties as a function of n? please clarify more formally the assumptions on b. also, please explain b.  please define the truncation kernel here.-- "we will provide a complementary analysis on the behaviour of the estimator for finite n."perhaps state a 'complementary theoretical analysis', i was led to believe you possibly meant only numerical, which of course, is much weaker.-- line 206, space missing-- remarks on thm 1: i would like more explanation of the relative size of the 3 biases.  the biases are a function of b, n, s and covariances.  some plotting showing the relative magnitude, say, of bias(San) vs bias(BC) would be very helpful.  for example, a heatmap showing bias(San)-bias(BC) for fixed n when varying b and s, or fixed function b_n and varying n & s.-- i don't understand the simulation setting.  please explain it more clearly, with equations, the notation for the 'parameter matrix' is unclear to me, what are '/' meant to denote? also, i don't know the abbreviation 'cmp'. if you are just trying to save space, i recommend removing some paragraph breaks, and keep content as clear as possible.-- a supplementary figure justifying footnote 4 is requested.-- "We average over R multivariate "  ok, what do you set R to be for these simulations?-- i think a better justification for *why* one would want to estimate a covariance matrix from an AR process, rather than the dynamics matrix, is in order.  in the end of the manuscript, you demonstrate an important application that totally justifies, but leading up to that, i was wondering.


Multivariate f -Divergence Estimation With Confidence
The problem of f -divergence estimation is important in the fields of machine learning, information theory, and statistics. While several nonparametric divergence estimators exist, relatively few have known convergence properties. In particular, even for those estimators whose MSE convergence rates are known, the asymptotic distributions are unknown. We establish the asymptotic normality of a recently proposed ensemble estimator of f -divergence between two distributions from a finite number of samples. This estimator has MSE convergence rate of O 1 T , is simple to implement, and performs well in high dimensions. This theory enables us to perform divergence-based inference tasks such as testing equality of pairs of distributions based on empirical samples. We experimentally validate our theoretical results and, as an illustration, use them to empirically bound the best achievable classification error.

Paper ID:  1260Title:  Multivariate $f$-Divergence Estimation With ConfidenceNOTE: Due to the short reviewing time and out of fairness to the other papers I was reviewing, I DID NOT do more than glance over the supplementary material.  In most cases, I therefore did not verify that results claimed by the authors are correct, but instead checked that they are plausible.Summary:  The authors consider an estimator for $f$-divergences based on the following procedure: given samples from two distributions $f_1$ and $f_2$, compute $k$-nn density estimators for Quality, clarity, originality and significance:  This paper proves a second-order result on estimators for information-theoretic/statistical functionals of pairs of distributions.  From a statistics/probability perspective this is an important result, since $k$-nn density estimators are relatively straightforward to compute.  The manuscript is not written in a way that is easy to read for the non-expert; the problem setup would be difficult to follow for the average NIPS attendee.  I therefore lowered my score.  The proofs (as determined from the sketches provided) are novel and appear to have broader applicability.Recommendation:  Overall, with some reorganization of the material with an eye towards helping the target audience (i.e., NIPS attendees) understand the results, this would be a nice addition to the program.  Without such reorganization, I would find it difficult to recommend acceptance.Comments:  (comments are numbered to allow for easy reference in the rebuttal)1)  In the related work, when discussing drawbacks of previous work, it might be nice to indicate which of the problems with that work the current result addresses, and which problems it does not address.2)  In the setup of the problem it would be best to be clear (for the reader) the space in which the estimators take their value.  For example, it does not seem that the results can handle vector-valued parameters, but perhaps I am mistaken.  A bit more mathematical precision/rigor earlier in the paper goes a long way.3)  The paper is more-or-less written with the expert in mind, which I find unfortunate.  The actual setup of the problem does not appear until the bottom of page 3 -- the estimator is given two sets of samples from $f_1$ and $f_2$ and must estimate a particular $f$-divergence.  It would *really* help the reader to have this setup first before talking about weighting ensembles of estimators etc. etc.4)  In the optimization before (1), what is the intuition behind the relaxation?5)  How is $N$ chosen?  6)  Why is the estimator in [1] defined using data X and Y and the plug-in estimator using X and X?  Is there a significance to the notation change?7)  When constructing the plug-in divergence estimator, is the estimate at $\mathbf{X}_j$ for $j \le N$ formed using the $k$-nn density estimate?  This is actually *not* specified in the manuscript as written.8)   The sketch of Lemma 4 is unintelligible to someone who is not going to read the proof (as opposed to the comments after Lemma 3) -- I would dramatically shorten it to make room for a more gentle introduction of the problem.  Just relegate the whole proof to the supplementary material.9)  The Broad Implications ("broader impacts?") in Section 3.2 seem appealing but also somewhat speculative.  While I appreciate the context, I am not sure this is the right location for these comments.  Perhaps move them to the conclusion?10)  In the conclusion: I realize that one must always put in some sort of "further work" to make it seem like the current result is building somewhere.  However, I am a little unconvinced that "deriving the nonasymptotic distribution of the estimator" is actually something that people will want to do.  To me the more convincing follow-up work would be the application of the main ideas of the proof here to other problems as outlined in section 3.2.11)  I didn't quite understand from the experiment section how the QDA classifier was trained -- perhaps a little more detail here would be appropriate -- there is still space in the paper to provide it, and it might give the reader who is not going to read the proof something concrete to look at.12)  Perhaps describing the estimator in an \begin{algorithm} \end{algorithm} environment would be helpful to the reader who wants to implement this procedure.  This may take up too much space.  However, it would make the result/approach more accessible.13)  One thing that is not at all clear from the description in Section 2 is that the index set $l$ is indexing over different neighborhood sizes for the $k$-nn density estimate.  Because a lot of the intuition/construction of the actual estimator is obscured by the author's writing, the overall idea of the proof is rather difficult to understand.Small things:- [Introduction] I think it's fairer to say that the idea of $f$-divergence generalizes several measures, rather than saying that many $f$-divergences have been defined -- it's a little more historically accurate.- there seems to be an overfull \hbox on page 5Please summarize your review in 1-2 sentences:  The authors prove the asymptotic normality of an estimator of $f$-divergence given two samples from two unknown distributions $f_1$ and $f_2$.  The estimator is formed by taking a plug-in estimate after kernel density estimation on each of the two distributions.Quality Score (1-10):  7Impact Score (1-2):  1Confidence (1-5):  4ADDITION AFTER AUTHOR FEEDBACK:  The detailed feedback from the authors has made me even more sanguine about this paper and I am revising my score upward to an 8.Quality Score (1-10):  8


Low Rank Approximation Lower Bounds in Row-Update Streams
We study low-rank approximation in the streaming model in which the rows of an n × d matrix A are presented one at a time in an arbitrary order. At the end of the stream, the streaming algorithm should output a k × d matrix R so that
, where A k is the best rank-k approximation to A. A deterministic streaming algorithm of Liberty (KDD, 2013), with an improved analysis of Ghashami and Phillips (SODA, 2014), provides such a streaming algorithm using O(dk/ ) words of space. A natural question is if smaller space is possible. We give an almost matching lower bound of Ω(dk/ ) bits of space, even for randomized algorithms which succeed only with constant probability. Our lower bound matches the upper bound of Ghashami and Phillips up to the word size, improving on a simple Ω(dk) space lower bound.

This paper investigates the lower bound of the space complexity for a specific streaming algorithm. In this streaming algorithm, the goal is to find a matrix R such that the matrix A can be approximated by A*R^\dag*R, where R^\dag is the pseudo-inverse of the matrix R. Also in the streaming model the rows of the matrix A are presented at a time in an arbitrary order.The problem investigated in this paper is interesting. This paper gives a lower bound of space needed to compute the matrix R. Compared with existing work, the improvement of the space lower bound is impressive. Since the main result of this paper is the improved lower bound, it is a very theoretical paper. It is not very easy for me to follow the whole paper. Many proofs in this paper are provided in the supplementary file. I went through this paper but did not check all details of this paper. But the proof of this paper sounds good to me.In terms of quality and significance, I think this paper is above the borderline of NIPS. My only concern is that many proofs are provided in the supplementary file, which makes reading this paper not easy. The readability can be further improved. Minor comment: the proof of Lemma 6 can be revised. For example, "expanding the square" can be removed from the long equation.


Parallel Double Greedy Submodular Maximization
Many machine learning problems can be reduced to the maximization of submodular functions. Although well understood in the serial setting, the parallel maximization of submodular functions remains an open area of research with recent results [1] only addressing monotone functions. The optimal algorithm for maximizing the more general class of non-monotone submodular functions was introduced by Buchbinder et al. [2] and follows a strongly serial double-greedy logic and program analysis. In this work, we propose two methods to parallelize the double-greedy algorithm. The first, coordination-free approach emphasizes speed at the cost of a weaker approximation guarantee. The second, concurrency control approach guarantees a tight 1/2-approximation, at the quantifiable cost of additional coordination and reduced parallelism. As a consequence we explore the tradeoff space between guaranteed performance and objective optimality. We implement and evaluate both algorithms on multi-core hardware and billion edge graphs, demonstrating both the scalability and tradeoffs of each approach.

Many submodular maximization could be optimized with decent approximation ratio by greedy-like algorithms. However, due to the inherently serial nature of the algorithms, they could not be easily parallelized. Leveraging the transaction processing model from database system, this paper presents two approaches to parallelizing the double greedy algorithm for unconstrained submodular maximization. In particular, the double greedy algorithm maintains two sets A and B where one can only add elements to A or removed elements from B. The authors consider each such operation an transaction just like the one in database system. The problem is then reduced to how to apply these transactions in parallel. The first approach (CF-2g) this paper proposes is a hogwild-style coordination-free algorithm where each thread runs locally with a possibly stale version of A and B. Due to the submodularity, the marginal improvement computed within each thread is always an upper bound of the true one. The authors also analyze the approximation ratio of CF-2g.The second approach (CC-2g) is a concurrency-control algorithm which guarantees the serializability and thus it shares the same approximation ratio as the serial double-greedy algorithm. It works by maintaining the upper and lower bound of A and B, which can be used to locally compute the upper bound and lower bound of the threshold in the serial algorithm. This "often" enables each thread to locally decide to add the element to A or remove it from B. And uncertain transactions are recomputed serially by the server. Figure 2 is a nice illustration of this process. As a consequence CF-2g promotes efficiency and CC-2g promotes solution quality. At the end, the authors conduct comprehensive experiments to evaluate the proposed algorithms. But seems like CC-2g algorithm is comparable with CF-2g in terms of speedup, and it even has more speedup than CF-2g in some cases (e.g. Fig. 3c) which is counter-intuitive. Is that due to some specific problem structure? I think the authors should give more discussions to make it more clear. Also, as pointed out in the last section, the delays might hurt the performance in distributed environment. But so far it's unclear empirically.


On Prior Distributions and Approximate Inference for Structured Variables
We present a general framework for constructing prior distributions with structured variables. The prior is defined as the information projection of a base distribution onto distributions supported on the constraint set of interest. In cases where this projection is intractable, we propose a family of parameterized approximations indexed by subsets of the domain. We further analyze the special case of sparse structure. While the optimal prior is intractable in general, we show that approximate inference using convex subsets is tractable, and is equivalent to maximizing a submodular function subject to cardinality constraints. As a result, inference using greedy forward selection provably achieves within a factor of (1-1/e) of the optimal objective value. Our work is motivated by the predictive modeling of high-dimensional functional neuroimaging data. For this task, we employ the Gaussian base distribution induced by local partial correlations and consider the design of priors to capture the domain knowledge of sparse support. Experimental results on simulated data and high dimensional neuroimaging data show the effectiveness of our approach in terms of support recovery and predictive accuracy.

This paper considers the problem of doing Bayesian inference when one would like to impose structural priors that restrict the support of the distribution.  It begins by motivating incorporating support constraints via the maximum entropy principle, where we would like to impose the constraint but deviate from some base distribution as little as possible, as measured by relative entropy. The distribution that optimizes this criteria is called the information projection of the base distribution onto the constraint set.  The paper begins by showing that this principle leads to intuitive results in the case of support restrictions (the information projection of p onto support set A is 1{x \in A} p(x) / \int_{x \in A} p(x) ), and that the notion can be used to describe Bayesian inference, where the posterior arises when the joint distribution is projected onto the support set defined by the observations.When we would like to perform an information projection onto the intersection of several sets, then the paper shows that this can be done by projecting onto each set in sequence in any order and the results are unchanged. This motivates an inference procedure in the special case of sparsity restrictions, which leads to the paper's main algorithm. The idea of the algorithm is to structure inference as an inside and outside optimization, where in the inner optimization, a posterior is computed where the set of k entries allowed to be nonzero are fixed, and in the outer optimization, the set of nonzero entries is optimized over. It is shown that this optimization problem can be formulated as a submodular optimization problem, which leads the authors to propose a greedy forward selection strategy for choosing nonzero dimensions. Results against a variety of baselines show the method to work quite well.The main pros of the paper are as follows:- the writing is very clear, and the arguments appear to be clearly justified; things seem technically correct- the formulation in terms of information projections is a clean and interesting way of formalizing things, providing arguments that help better understand what it means to add constraints to models- the algorithm appears to work wellThe main con is that there is a lot of technical work to get to results that are not terribly surprising. From a practical perspective, I'm not convinced that Section 2 of the paper is necessary to get to the algorithm in Section 3. It seems that we could have simply observed that the sparsity constraints can be viewed as a union of simple constraint sets, then arrived fairly naturally at the objective that is used in Section 3, and then derived the algorithm in the paper. The fact that J(s) is submodular is easy to see. So while I enjoyed reading the paper, I'm not sure I left with any new tools for imposing prior knowledge in my models.