Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
We develop a general duality between neural networks and compositional kernel Hilbert spaces. We introduce the notion of a computation skeleton, an acyclic graph that succinctly describes both a family of neural networks and a kernel space. Random neural networks are generated from a skeleton through node replication followed by sampling from a normal distribution to assign weights. The kernel space consists of functions that arise by compositions, averaging, and non-linear transformations governed by the skeleton's graph topology and activation functions. We prove that random networks induce representations which approximate the kernel space. In particular, it follows that random weight initialization often yields a favorable starting point for optimization despite the worst-case intractability of training neural networks.

The authors introduce the notion of computational skeleton and show how a skeleton can be related to a RKHS (more precisely, to a compositional kernel) through the concept of "dual activation".  They also introduce a class of neural networks, called "realisation of a skeleton", defined through a given number of replications of a computational skeleton and a given number a of outputs (r and k, respectively).  The main results of the paper consist in analysing in what sense the representation generated by a random initialisation of such a neural network (i.e., a skeleton realisation) approximates the compositional kernel related to the underlying skeleton.
# Some typos/remarks:  -At line 154, do the authors mean "any continuous PSD function"? -Two typos at lines 171 (y instead of x') and 177 ("of in the induced space").  -I am not sure about the meaning of the "2-norm" mentioned at lines 173 and 184, maybe the author should specify it.  -At line 190, S is a computational skeleton, and k_{S} is a compositional kernel.  -In Lemma 11 (appendix), I do not really understand what does b_{|A|} mean.    #Questions (maybe not founded...):  -Is it possible to interpret the fact that "for a random activation, the empirical kernel k_w approximates the kernel k_S", as a kind of "law of large numbers", in the sense: "the replication of the computation skeleton counterbalances the fact that the weight of the network are random"?   -What kind of impact the operation of "learning the weights of the network" could have, for instance, on the relation between k_w and k_S (and on the related Hilbert spaces)?   -Is it possible, in view of Thm. 5 and 6, to draw a parallel between "learning with a skeleton realisation" and "learning with a compositional kernel"?   ########################################### ### After rebuttal  My major concern about the paper is the lack of discussion on "why these results may be important or interesting". In fact, there is almost no discussion at all about the implication of the results of the paper.   The notions of "computational skeleton" and "realisation of a skeleton" seem relatively interesting and aim at generalising the kernel construction proposed in [13] and [29]. This "construction" is in my view the main contribution of the paper.   On a theoretical point of view and from my understanding, the main results of the paper are Thm 3. and 4. However, these results are not "so surprising", since, as confirmed by the authors, they can be interpreted as a kind of "law of large number": the random activation of the network is "counterbalanced" by the replication of the skeleton. Note that there is a big typo in eq. (12) of the supplementary material. As indicated by Table 1, some results of this type are already known.   From my understanding, Thm 5 and 6 are in fact just "advanced corollaries" of Thm 3. and 4 (as indicated at line 586 of the appendix, with a typo).  In their feedback, the authors indicate (about Thm 5 and 6) that: "this shows that training the final layer is essentially equivalent to learning with the corresponding kernel". However, for "kernel-based learning" (line 82), the authors deal with the classical "loss + regularisation" formulation, while classically, "only loss" is considered for "Neural Network-based learning" (line 74); so that I do not really understand how "training the final layer" could be sensitive to "the size of the ball in the RKHS".  In my view and from my understanding, these results are about "the difference between learning with K_S or K_w in the classical framework of kernel-based learning", so that I do not really agree with the answer of the authors.   Another "drawback" of the approach is, in my opinion, that the input space is restricted to be of "hypersphere-type".    Finally, the paper is quite difficult to read and its construction may, in my opinion, be improved in order to try to clarify and simplify the paper.


Learning the Number of Neurons in Deep Networks
Nowadays, the number of layers and of neurons in each layer of a deep network are typically set manually. While very deep and wide networks have proven effective in general, they come at a high memory and computation cost, thus making them impractical for constrained platforms. These networks, however, are known to have many redundant parameters, and could thus, in principle, be replaced by more compact architectures. In this paper, we introduce an approach to automatically determining the number of neurons in each layer of a deep network during learning. To this end, we propose to make use of a group sparsity regularizer on the parameters of the network, where each group is defined to act on a single neuron. Starting from an overcomplete network, we show that our approach can reduce the number of parameters by up to 80% while retaining or even improving the network accuracy.

Group lasso regularization was added to remove neurons for deep model selection. Reported experimental results on a couple of deep models showed the expected model reduction with comparable prediction accuracy on several benchmark datasets.
Stochastic proximal gradient descent was presented to solve the deep model training with the proposed model selection based on group lasso regularization. However, the implementation details were not presented, especially regarding the iterative updates for the "loss-based gradient  step". From the context, it seems that the iterative updates in the original deep model training procedure were simply modified based on Eq. 5. If that is the case, the claim of the generality of the method for "general deep networks" as it will depend on the optimization procedure of the deep model training.   If the motivation of the presented work is to remove redundant neurons. it appears natural to compare the proposed method with the dropout method to see the performance. Without that, it is difficult to evaluate the significance and advantages of the method.   The presented performance evaluation is problematic. It is somewhat strange that both the performance evaluation results were done based on some modifications of the original deep models, either removing the first two layers (BnetC) or changing maxout to maxpooling (Dec3). Why was the comparison not done to the original model without changes, especially considering the reported decrease of top-1 accuracy in Table 1? It is not described how "accuracy gap" was computed. If it is simply the top-1 accuracy difference, it does not seem to be consistent based on the tables in Figs 3 and 4.  More critically, as typically done in sparse models, the sensitivity of the method with respect to penalty coefficients should be analyzed, especially with the complicated loss function in deep models. There are also many confusing places in experiments. For example, in Table 1, what is the difference between BnetC and "Ours-BnetC" and for other similar pairs? What does "S-GS" stand for (SGL?) in the table of Fig. 4?   There are also typos throughout the paper. For example, in line 275: "zeroed-our" should be "zeroed-out"; in line 293, "were" should be "where"?


Temporal Regularized Matrix Factorization for High-dimensional Time Series Prediction
Time series prediction problems are becoming increasingly high-dimensional in modern applications, such as climatology and demand forecasting. For example, in the latter problem, the number of items for which demand needs to be forecast might be as large as 50,000. In addition, the data is generally noisy and full of missing values. Thus, modern applications require methods that are highly scalable, and can deal with noisy data in terms of corruptions or missing values. However, classical time series methods usually fall short of handling these issues. In this paper, we present a temporal regularized matrix factorization (TRMF) framework which supports data-driven temporal learning and forecasting. We develop novel regularization schemes and use scalable matrix factorization methods that are eminently suited for high-dimensional time series data that has many missing values. Our proposed TRMF is highly general, and subsumes many existing approaches for time series analysis. We make interesting connections to graph regularization methods in the context of learning the dependencies in an autoregressive framework. Experimental results show the superiority of TRMF in terms of scalability and prediction quality. In particular, TRMF is two orders of magnitude faster than other methods on a problem of dimension 50,000, and generates better forecasts on real-world datasets such as Wal-mart E-commerce datasets.

This paper presents an approach to improve matrix factorization framework in the context of forecasting, where one of the matrix dimensions is time. The main contribution is the proposal of a regularization term that is the log-likelihood of a temporal model regarding some parameters to estimate. In fact, this is a funny way to put it. What their model is really doing is generating n time series by linearly combining k latent time series, governed by a Gauss-Markov model, and where the n weights of the linear combinations can be learned.  This approach has several benefits, first it can handle missing-value easily, second it can share information across different time-series while keeping temporal coherency in the model. Contrarily to graph-based regularization, the weights of edges do not have to be specified but can be learned and negative correlation are supported. The author instantiate this framework with a simple AR model. In this special case, they show how the temporal regularizer can be expressed as graph regularization which allow to use common solvers. Finally, they do comparison on 5 datasets with improvement in running time and around 10% improvement in accuracy compared to other methods.
The paper is well written (except maybe I find the formulation of the time series part as "regularizer" overly complex, see below), and also well motivated. The proposed method does better than a range of competitors in experiments on a set of datasets, some of which are proprietary (which is fine with me). Finally, I find the application domain (retail forecasting) exciting and of high impact, and would like to see more work in the NIPS community in this direction (even though the lack of publicly available forecasting datasets of sufficient quality is clearly a hindrance).  On the other hand, I am somewhat disappointed by the overall scope of the work. In fact, what is proposed here is to model a smaller set of k dependent time series in a standard fashion (correlated AR), and then map these to n (partially) observed series using n linear weight vectors. That is all. Phrasing this as "regularization" is uncommon and does not help the understanding. I am not an expert on work in this area, but find it hard to accept that something like this has not been attempted before.  There is no mentioning of the fact that demand forecasting distributions are typically intermittent, and therefore poorly modelled by Gaussian noise (I recall an ICML paper by Chapados a few years ago that made this point). Also, there is no mentioning of basic structural elements, such as seasonality, promotions, or other features. For example, Walmart will sell more just before Christmas, and more turkey before Thanksgiving. In fact, what is described here, does not depend on other data than the demand targets at all. I doubt that a method like this would work very well on real demand data.  The second problem is the range of methods the authors compare against. Given that the proposed method is not very realistic as far as forecasting goes (no seasonality, no external inputs such as promotions, ...), it is understandable the authors compare against equally simple methods. But why not compare against simple things like Hyndman's forecast package, run separately on each series? This would take seasonality factors into account. The authors also do not try other basic setups in the ARIMA space. In fact, I am not convinced from these results that MF would in fact outperform an approach of individually forecasting each series, if only this was done with best practices. I have the strong feeling that a paper with this kind of evaluation would not pass at a forecasting conference or journal.  It also seems a bit unfair in my opinion to use only AR(1) in the comparison whereas domain knowledge was used to specify the lags of TRMF-AR. From the running time mentioned in Figure 5, at least AR({1,24}) for hourly grain and AR({1,52}) for weekly grain experiments can be used.   The scalability argument given by the author versus AR is also contrasted by the fact that AR is embarrassingly parallel and can be trained in multiple machine without difficulties allowing to scale to arbitrary large number whereas the proposed method does not enjoy this property.  Detailed comments:  p5: “unlike most approaches mentioned in section 2.2”. This sounds a bit vague to me, probably this refers only to Graph based approach. However in the second point, it says that the methods cant handle different lag which is not true for Graph based methods.   p6: please add Coefficient of Variation in the table to account for the forecast difficulty in Table 1.  p6: About Walmart dataset, the missing ratios indicated in Table 1 are not the initial one. The appendix mentions that time-series are aligned by adding unobserved data points. Please also add the proportion of unobserved data in the main document of the initial time-series, the proportion is probably much smaller and this gives the wrong impression if the reader does not check the appendix (if this ratio is very small then all methods could be tested in table 2 by replacing unobserved by zeros).  p7: although it can be understood in next page with the lag chosen for electricity and traffic datasets, it will help to specify the hourly time grain sooner, it is only stated explicitly in the appendix.  p7: Figure 4. the values of n (500, 1000, 5000, 10000, 50000) are not easy to read. W  p8: R-DLM should could be removed as it cannot handle any of the datasets except the articial one (maybe just mention that it only handles n lower than 32 in the text).


Adaptive Concentration Inequalities for Sequential Decision Problems
A key challenge in sequential decision problems is to determine how many samples are needed for an agent to make reliable decisions with good probabilistic guarantees. We introduce Hoeffding-like concentration inequalities that hold for a random, adaptively chosen number of samples. Our inequalities are tight under natural assumptions and can greatly simplify the analysis of common sequential decision problems. In particular, we apply them to sequential hypothesis testing, best arm identification, and sorting. The resulting algorithms rival or exceed the state of the art both theoretically and empirically.

The authors obtain Hoeffding type inequalities for stopped sequences. The classical Hoeffding inequality is a decaying exponential bound on the probability of a rare events for a sum of iid random variables. But this inequality is valid for a fixed number of random variables. The authors claim that it is of interest to obtain similar inequalities for random number of random variables. The authors appear to succeed in obtaining such an inequality (Thm 1 and Cor 1). The bound is valid for subgaussian random variables. The authors then claim that they can use this inequality and the idea behind it to obtain competing algorithms for some popular problems in statistics and machine learning (MAB etc, Thm 5).
The paper is very well written, is easy to follow, and the contributions are stated clearly and the problem is well motivated. I think the paper is a good contribution to the literature. I do have few comments that the authors may use to improve the paper.   1. Algorithm 1: The way it is written makes the concepts confusing. f_\alpha is defined such that its value depends on the knowledge of alpha^*. The latter is not known. So, how do you "draw a sample from the arm with largest value of f_\alpha", and also how do you implement the next line: remove from A ...? You cannot compare the inequality because alpha^* is not known. Perhaps the authors have made a mistake here in terms of notations?   2. Defintion 4 should be stated as an Algorithm?   3. The authors may wish to add few lines on how their inequalities obtained are not just a rewording of the law of iterated logarithm (LIL), or is not just a simple implication. I mean if you provide some intuition on the nontrivially of these results given LIL, it would make the paper complete.


Active Nearest-Neighbor Learning in Metric Spaces
We propose a pool-based non-parametric active learning algorithm for general metric spaces, called MArgin Regularized Metric Active Nearest Neighbor (MARMANN), which outputs a nearest-neighbor classifier. We give prediction error guarantees that depend on the noisy-margin properties of the input sample, and are competitive with those obtained by previously proposed passive learners. We prove that the label complexity of MARMANN is significantly lower than that of any passive learner with similar error guarantees. Our algorithm is based on a generalized sample compression scheme and a new label-efficient active model-selection procedure.

This paper studies active learning for nearest neighbor classifiers in metric spaces. Built upon a series of work on nearest neighbor classifiers (e.g. Gottlieb and Kontorovich,  2015), the paper provides an active learning algorithm for nearest neighbors, dubbed MARMANN, that has nontrivial label complexity guarantees. Specifically, an example is provided to show that the proposed algorithm has superior label complexity over its passive version.  The key technical trick is to select an appropriate "scale", t, such that the margin error term epsilon(t), and the sample compression generalization bound phi(t), are almost of the same order. After selecting the scale, a 1-NN classifier is output.
Technical quality: the proofs in this paper are quite sound.   For practical implementation, I wonder if the labels queried for different scales t can be reused. Perhaps we can organize the data in some tree structure, e.g. cover trees (Beygelzimer, Kakade, Langford, 2004), and walk down the cover tree in some way.  In addition, I wonder if the idea of "active comparison of pairs of classifiers" in (Balcan, Hanneke, Vaughan, 2010) can be used in the model selection (scale selection) procedure, since the term O(1/\hat{G}) term is a bit unsatisfying to me.   The fact that the label complexity bound is exponential in the doubling dimension of the metric space is a bit concerning, although I think this type of dependency may be difficult to get rid of for nonparametric classifiers.  Novelty: although the general idea of using sample compression bounds for condensed nearest neighbor classification has appeared in e.g. (Gottlieb, Kontorovich, Nisnevich, 2014), the idea here using random sampling in each Voronoi cell and noisy binary search the "correct" scale to minimize the error bound is novel.  Potential Impact: There has been relative little literature on active learning with non-parametric classifiers, e.g. nearest neighbor classifier. I think this is a nice starting point and will spur future work.  Clarity and Presentation: The main result, Theorem 3.1 is a bit difficult to appreciate on first sight, since it is very different from PAC learning with a fixed/parameterized hypothesis class.   In addition, the paper may need some explanation on how close is the error of the classifier to the Bayes optimal, in additive sense.  I find Theorem 5.4 and 5.5 difficult to follow, but I understand the high level idea as follows -- we would like to find two neighbor scales t_0 < t_0' such that \hat{\epsilon}(t_0) < \phi(t_0) \hat{\epsilon}(t_0') > \phi(t_0') If this is true then the scale t_0 is optimal up to constants in terms of error bound achieved. Is that right?   *** UPDATE AFTER READING AUTHOR FEEDBACK AND DISCUSSION *** I think overall this is a nice paper that provides active learning algorithm working in general metric space and providing nontrivial label complexity guarantees. I vote for acceptance.   In the author feedback, the author says that in realizable case, Algorithm 1 of BHV10 is not directly applicable; I can roughly see that since the number of scales to be considered is O(m^2). But under some standard regularity assumptions (density for each class does not change too fast), can one use a "doubling trick", i.e. set scales t_L = t_1 \leq t_2 \leq .. \leq t_n-1 \leq t_n = t_U  such that for any i,  nu(t_i+1) \leq 2 * nu(t_i)  phi(t_i-1) \leq 2 * phi(t_i) (*)   and we make sure that  nu(t_U) \leq 1/m  phi(t_L) \leq 1/m (*)   Now consider running Algorithm 1 of BHV10 with a label budget L. for each scale t, we call GenerateNNSet(t, I, delta) with label budget L/n. (If during the execution of the procedure it runs out of label budget then it simply returns a trivial classifier.) Then active comparison is performed to output a final classifier. This can make sure that the algorithm adapts to the "correct" scale with a additional multiplicative factor of O(n log n/\delta) label complexity.   Admittedly, my idea is a crude one, since finding scales that satisfies conditions (*) apriori is a bit difficult without knowing anything about the data distribution.  (Reviewer 1 agrees that something like this might work, but some technical details need to be worked out.)


Direct Feedback Alignment Provides Learning in Deep Neural Networks
Artificial neural networks are most commonly trained with the back-propagation algorithm, where the gradient for learning is provided by back-propagating the error, layer by layer, from the output layer to the hidden layers. A recently discovered method called feedback-alignment shows that the weights used for propagating the error backward don't have to be symmetric with the weights used for propagation the activation forward. In fact, random feedback weights work evenly well, because the network learns how to make the feedback useful. In this work, the feedback alignment principle is used for training hidden layers more independently from the rest of the network, and from a zero initial condition. The error is propagated through fixed random feedback connections directly from the output layer to each hidden layer. This simple method is able to achieve zero training error even in convolutional networks and very deep networks, completely without error backpropagation. The method is a step towards biologically plausible machine learning because the error signal is almost local, and no symmetric or reciprocal weights are required. Experiments show that the test performance on MNIST and CIFAR is almost as good as those obtained with back-propagation for fully connected networks. If combined with dropout, the method achieves 1.45% error on the permutation invariant MNIST task.

Feedback alignment (FA) is a recently developed method for training neural networks which is more biologically plausible than backpropagation (BP. Unlike BP, where the error signal is propagated on the backward pass through the transpose of the forward weights (which is difficult to square with our understanding of neural constraints), in feedback alignment a random weight matrix is used to propagate the error. Surprisingly, propagating the error though random weights works well, learns quickly (although sometimes slightly slower than BP) and may even act as a regularizer and outperform backpropagation in some situations.  This paper proposes, analyses and experimentally tests a variant of this approach they term "direct feedback alignment." In this scheme, rather than propagating the error through random weights at each layer, the error signal for each layer is derived through random ways directly from the output layer. The find that this scheme is able to perform similar to FA, and unlike both BP and FA is  capable of training very deep networks.
This paper is relatively well-written and supports its claims. It develops a reasonable analysis of the direct feedback alignment, although it follows closely from the original feedback alignment work. For this reason, it would probably be possible to abbreviate parts of the analysis without much loss to the reader.  The primary weakness of this work is impact. The idea of "direct feedback alignment" follows fairly straightforwardly from the original FA alignment work. Its notable that it is useful in training very deep networks (e.g. 100 layers) but its not clear that this results in an advantage for function approximation (the error rate is higher for these deep networks). If the authors could demonstrate that DFA allows one to train and make use of such deep networks where BP and FA struggle on a larger dataset this would significantly enhance the impact of the paper.  In terms of biological understanding, FA seems more supported by biological observations (which typically show reciprocal forward and backward connections between hierarchical brain areas, not direct connections back from one region to all others as might be expected in DFA). The paper doesn't provide support for their claim, in the final paragraph, that DFA is more biologically plausible than FA.  Minor issues: - A few typos, there is no line numbers in the draft so I haven't itemized them. - Table 1, 2, 3 the legends should be longer and clarify whether the numbers are % errors, or % correct (MNIST and CIFAR respectively presumably). - Figure 2 right. I found it difficult to distinguish between the different curves. Maybe make use of styles (e.g. dashed lines) or add color. - Figure 3 is very hard to read anything on the figure. - I think this manuscript is not following the NIPS style. The citations are not by number and there are no line numbers or an "Anonymous Author" placeholder. - I might be helpful to quantify and clarify the claim "ReLU does not work very well in very deep or in convolutional networks." ReLUs were used in the AlexNet paper which, at the time, was considered deep and makes use of convolution (with pooling rather than ReLUs for the convolutional layers).


Optimal Learning for Multi-pass Stochastic Gradient Methods
We analyze the learning properties of the stochastic gradient method when multiple passes over the data and mini-batches are allowed. In particular, we consider the square loss and show that for a universal step-size choice, the number of passes acts as a regularization parameter, and optimal finite sample bounds can be achieved by early-stopping. Moreover, we show that larger step-sizes are allowed when considering mini-batches. Our analysis is based on a unifying approach, encompassing both batch and stochastic gradient methods as special cases.

This paper establishes optimal rates (up to log factors here and throughout this review) using multi-pass mini-batch SGM. Most results hold with moderately high probability (failure probability $1  - 1/m$), and it appears that all the guarantees are valid only for $m$ ``large enough'' (greater than $m_0$), which might not be known without additional information related to the capacity.  In the case of b = 1 (non-mini-batch), optimal rates are obtained by a method which depends on the capacity only through the stopping time (Corollary 3.3). Corollary 3.5 appears to be suboptimal; it is similar to Corollary 3.5 but involves taking $b = \sqrt{m}$. Corollaries 3.6 and 3.8 have stopping times ostensibly independent of the capacity, but either the learning rate (Corollary 3.6) or the batch size (Corollary 3.8) depend on the capacity. Corollary 3.9 is similar to Corollary 3.3 in the sense of providing a guarantee for a method that depends on the capacity only through the stopping time, but this later result now uses $b = m$, corresponding to a batch-like multi-pass SGM method.  Some experimental results exhibit the expected behavior from the bias and sample error.
This work provides a strong contribution in that it apparently is the first work to net optimal rates (up to log factors) for SGM, and moreover, it also handles a mini-batch analysis which includes the (full) batch method as a special case. Such rates previously had been established only for the (batch) ridge regression method.  My interpretation of what all the results actually show is given in the Summary.  I find the current solution of relying on cross-validation for adaptation to be a bit of an inelegant cop-out (even if there is a theoretically-supported method for using it); given that several of your corollaries provide a guarantee where $\zeta$ and $\gamma$ enter the picture only through $T^*$, can you provide a self-monitoring method that decides when to stop? In particular, I find the most exciting results to be Corollaries 3.3 and 3.9, as only the stopping time depends on the (unknown) capacity parameters, and so such an online stopping mechanism might be possible.  An issue with all of the results is that they are only valid for ``large enough'' $m$, which cannot be verified without knowing the unknown capacity parameters. I am somewhat willing to overlook this weakness, however, as it may be unavoidable.  It would have been useful if the authors briefly sketched, in the main paper, how they proved bounds on each of the three components in their error decomposition. Even a few sentences for each would suffice (given the space constraints, much more may be tricky). As I explain below, this will help both with the clarity of the technical portion and the impact of this work. I did not actually get much out of the experimental results near the end, so I would argue for partially cutting them in favor of giving readers more information about the technical novelty of your results and particular sub-results that might be reused in other works. This could help boost the impact of your work. At a minimum, the current way the Appendix reads is somewhat adversarial. For starters, the appendix is not searchable. It appears that it is just a sequence of images, which is rather annoying. Using 'pdfjam' in the future should lead to a proper splitting of PDFs without sacrificing the text-encoding. But most importantly, if you provide proof sketches in the appendix, then readers will have some idea of the overall argument, and this will help serve as a guide to the proofs. This will help with technical reuse so that other works can continue in this line. In its current state, the appendix is more or less a barrage of proofs without much indication of the overall strategy (I mean this constructively!). I therefore found the proofs very hard to check (although I read some of the initial proofs and they appear to be correct), and this certainly correlates to the results being hard to adapt.


The Power of Optimization from Samples
We consider the problem of optimization from samples of monotone submodular functions with bounded curvature. In numerous applications, the function optimized is not known a priori, but instead learned from data. What are the guarantees we have when optimizing functions from sampled data? In this paper we show that for any monotone submodular function with curvature c there is a (1 c)/(1 + c c 2 ) approximation algorithm for maximization under cardinality constraints when polynomially-many samples are drawn from the uniform distribution over feasible sets. Moreover, we show that this algorithm is optimal. That is, for any c < 1, there exists a submodular function with curvature c for which no algorithm can achieve a better approximation. The curvature assumption is crucial as for general monotone submodular functions no algorithm can obtain a constant-factor approximation for maximization under a cardinality constraint when observing polynomially-many samples drawn from any distribution over feasible sets, even when the function is statistically learnable.

Optimization from samples of monotone submodular functions with bounded curvature is studied. The authors provide and prove hard bounds on the approximation ratio for this problem. Specifically, using a probabilistic argument, they show that the approximation bounds are tight. They focus on the constrained case and consider the problem of maximizing the function with a cardinality constraint and provide a simple algorithm to solve the problem. Simulation study suggests that the algorithm is better than the best sample and a random set.
The paper is overall well written, in particular: section 1. sets up the problem well; intuition for the proofs were helpful in checking them and I think it should be accessible to readers with some basic understanding of submodular optimization literature. The references given by the authors are standard, good and popular which is good. In particular, I liked the hardness section where the idea is to classify elements of the ground set into two categories: good, bad and poor. Poor elements are only useful for making the constructed function monotone submodular whereas the good and bad are elements that carry very less "information". I am not sure if information is the right word here since usually it is related to entropy functions, it will be nice if the authors can describe why they call it that way.   One major concern that I have is the simulation experiments. In general, I think  that a single random set is not fair for comparison, if you are going to compare it with randomized selection, I think many draws should be conducted for evaluation (similar to boot strapping) and a confidence interval or bar would convey more than a single random draw. Secondly, while this seeme like a theoretical paper, nonetheless, I feel like it can/should be supplemented with some experiments using real datasets on problems that the authors mention in section 1. This is the main reason for my score for question 5, 7 (technical quality, Potential impact or usefulness).  While the analysis works for the special case of cardinality constrained problems, any thoughts on how to adapt the analysis to other constraints like cover or knapsack constraints? (I suppose that this is not simple as it sounds and can be highly nontrivial?)


SURGE: Surface Regularized Geometry Estimation from a Single Image
This paper introduces an approach to regularize 2.5D surface normal and depth predictions at each pixel given a single input image. The approach infers and reasons about the underlying 3D planar surfaces depicted in the image to snap predicted normals and depths to inferred planar surfaces, all while maintaining fine detail within objects. Our approach comprises two components: (i) a fourstream convolutional neural network (CNN) where depths, surface normals, and likelihoods of planar region and planar boundary are predicted at each pixel, followed by (ii) a dense conditional random field (DCRF) that integrates the four predictions such that the normals and depths are compatible with each other and regularized by the planar region and planar boundary information. The DCRF is formulated such that gradients can be passed to the surface normal and depth CNNs via backpropagation. In addition, we propose new planar-wise metrics to evaluate geometry consistency within planar surfaces, which are more tightly related to dependent 3D editing applications. We show that our regularization yields a 30% relative improvement in planar consistency on the NYU v2 dataset [24].

See below.
The paper proposes a method for recovering scene geometry from a single RGB image. This method uses a dense CRF with terms that enforce consistency between point-wise depth and normal estimates, using regularizers based on classification of planarity and presence of depth boundaries. Each of these estimates (depth, normal, planarity, edges) comes from a separate network proposed for each task in prior work. Experimental evaluation show modest improvement over baseline depth and normal prediction models.  In addition to the geometry-terms in proposed DCRF-based model, the paper's contributions include using multiple passes through the depth and normal networks with dropout to derive 'confidence' values of these metrics, and joint training to fine tune the depth and normal networks.   While significantly engineered for its specific application domain, the paper does demonstrate a successful example of inference with a regularized objective, where different terms are predicted from trained neural networks. I believe this will be of interest to many in the NIPS audience.  However, the overall presentation of the method---including some of the motivation and evaluation---leaves some questions un-answered:  - I have to say the benefits from joint training don't appear to be significant---could this improvement be explained by essentially additional training iterations ? Perhaps, a better test would've been to train these networks from scratch, or perhaps further train the depth and normal networks for an equal number of further iterations on the same training set (but with direct losses instead of post-DCRF losses).  - The fact that Eigen-VGG(JT) does better than the vanilla version is a little surprising---the joint training optimizes each network to perform better within the DCRF with complimentary information from the others. Why would it improve stand-alone performance ? Surely the direct loss used in the original training of each of these networks is the right choice for stand-alone performance. This again makes me believe that the improvement seen due to joint-training is because of the extra training iterations, and not because of the "joint" loss.  - Why were confidences not used during joint training ? One could "tie together" multiple forward passes for the same example, and use that to derive confidence values, and backprop through these. Is it that the extra forward passes weren't worth the slower training time ? It is also worth reporting results for DCRF (JT)---i.e., DCRF results without confidence values, but with joint training---since this corresponds to using the network in the same regime for which it was trained.  - While most of the terms in the DCRF objective "make sense", they are hand-crafted and involve making a lot of design decisions. Given the small size of the NYUv2 test set (and that its GT data is available), it is worth mentioning whether all these design decisions were made on the val set. Along similar lines, how did authors decide on the parametric form for dropout-variance to confidence ? Was this also determined on the val set from [9] ?  - The work of [28] is perhaps the closest to the proposed method, since it also uses a dense CRF with joint terms for depth and edges. I think the paper could use a more thorough discussion of it. [28]'s results are worse than those reported in this paper, but that may be because of the VGG-trained [6] baseline network used by this paper. In fact, [28] appears to show a more significant relative improvement between its depth-only and depth+edge CRFs (than this paper's DCRF-conf(JT) vs [6]). A fair question to ask is how would [28]'s "HCRF" do with [6]'s depth network and VGG-pretraining.  - The paper could do with a formal definition of 'planarity' means in the P0 output---specifically, at what resolution / region-size is point-wise planarity defined (in the limit, every infinitesimal surface point is a plane!) ? Is a planar region defined as one with some minimum number of contiguous pixels in which normals are constant ?  - The baseline errors for [6] appear to be different from those reported in [6] itself. Could the authors comment ? (the biggest difference appears to be in the RMSE errors---are they computing means of mean(per-image-RMSE), or sqr-root(MSE across all images). I believe the latter is the convention used to report results on NYUv2.  ===Post-rebuttal The authors have largely addressed the questions in my review. I'd say the paper just about meets the threshold for publication at NIPS (with the promised additional discussions for clarity).  I'd also recommend that authors consider doing joint training with the RAW distribution of the NYUv2 set (rather than just the smaller official release). I think the current experiments might be underselling the utility of the method because of the limited training data.


Fast Mixing Markov Chains for Strongly Rayleigh Measures, DPPs, and Constrained Sampling
We study probability measures induced by set functions with constraints. Such measures arise in a variety of real-world settings, where prior knowledge, resource limitations, or other pragmatic considerations impose constraints. We consider the task of rapidly sampling from such constrained measures, and develop fast Markov chain samplers for them. Our first main result is for MCMC sampling from Strongly Rayleigh (SR) measures, for which we present sharp polynomial bounds on the mixing time. As a corollary, this result yields a fast mixing sampler for Determinantal Point Processes (DPPs), yielding (to our knowledge) the first provably fast MCMC sampler for DPPs since their inception over four decades ago. Beyond SR measures, we develop MCMC samplers for probabilistic models with hard constraints and identify sufficient conditions under which their chains mix rapidly. We illustrate our claims by empirically verifying the dependence of mixing times on the key factors governing our theoretical bounds.

This paper presents and analyzes several Markov chains for sampling from various discrete probability distributions that allow for hard constraints. The authors analyze the mixing rate of the bases-exchange Gibbs sampler and the add-delete Gibbs sampler when the constraints are special types of matroid bases.   The authors then go on to show that a modification of the bases-exchange Gibbs sampler can be shown to mix rapidly when the target distribution satisfies the condition of being Strongly Rayleigh, a condition which is satisfied by Determinantal Point Processes.  Finally, experimental results are given which empirically demonstrate that the theoretical bounds presented depend on the correct quantities.
This paper is technically sound and addresses interesting questions relating to sampling and mixing rates. In fact, this paper generalizes two previous results:  (a) Section 2 can be considered an extension of Gotovos et al. (NIPS 2015). Unlike that paper, however, this paper allows for hard constraints. It is quite remarkable that even with these hard constraints, the authors are able to recover the same mixing rates as Gotovos et al.  (b) Section 3 can be thought of as an extension of Anari et al. (COLT 2016), but the difference is that Anari et al. could only handle homogeneous Strongly Rayleigh distributions, while this paper can handle any Strongly Rayleigh distribution. The proof, in which a Strongly Rayleigh distribution is viewed as the marginal distribution of a homogeneous Strongly Rayleigh distribution over a larger set of elements, is quite clever.    As best as I can tell, all the statements given in the paper and all the final bounds in the proofs presented in the supplementary material are essentially correct, modulo some constant factors. HOWEVER, there are some grave typos and undefined notation which make the intermediate steps in the proofs unintelligible.    Detailed comments:  1. In the proofs for Theorem 2, it appears that a factor of 2 is missing due to the fact that they are using the lazy version of the Markov chain. Thus, both of the mixing rate bounds should be increased by a factor of 2.  2. In the equations following lines 340 and 370, the factor of Z should appear in the denominator and not the numerator. Fortunately, this appears to be a typo and does not propagate through the rest of the proofs.  3. In equation (2.3) and all of appendix A, every occurrence of exp(\beta(F(\sigma_S(X,Y)) + F(\sigma_T(X,Y)))) should be replaced with exp(\beta F(\sigma_S(X,Y))) + exp(\beta F(\sigma_T(X,Y))). Similarly, every occurrence of exp(\beta(F(U_S) + F(U_T))) should be replaced with exp(\beta F(U_S)) + exp(\beta F(U_T)). This too appears to be a typo and does not propagate through the rest of the proofs.  4. In appendix A.2 and A.3, the quantity Z_F is never defined, although it seems to be taken as \sum_{U_S} exp(\beta F(U_S)).   5. In Theorem 4, it ought to be made explicit that \alpha_1 and \alpha_2 are functions of edges (S,T).  6. In Theorem 4 and the proof in appendix B, the notation |  |_+ and (  )_+ ought to be defined.  7. In the proof of Theorem 5, it should to be made clear that Theorem 10 is being instantiated with N := 2N and k := N.  8. In the figures of section 4 and appendix D, the meaning of the dashed lines ought to be made clear.


Optimal Black-Box Reductions Between Optimization Objectives *
The diverse world of machine learning applications has given rise to a plethora of algorithms and optimization methods, finely tuned to the specific regression or classification task at hand. We reduce the complexity of algorithm design for machine learning by reductions: we develop reductions that take a method developed for one setting and apply it to the entire spectrum of smoothness and strong-convexity in applications. Furthermore, unlike existing results, our new reductions are optimal and more practical. We show how these new reductions give rise to new and faster running times on training linear classifiers for various families of loss functions, and conclude with experiments showing their successes also in practice.

The authors consider the composite convex optimization problem of minizing the sum of a smooth and a simple convex function, which prox operator is efficiently computatble. The analysis of the authors is based on an abstract decrease property of descent methods for smooth strongly convex optimization: the HOOD property. The main mechanisms behind the proposed method is to approximate convex problems by smooth strongly convex models and tune the level of approximation as a function of the iteration counter. The authors propose and analyse nested algorithms, the inner loop is a call to an abstract descent method which satisfies the HOOD property and the outer loop consists in adjusting the smoothness and / or strong convexity levels of approximate objective function. Careful tuning of approximate model parameters and inner loop calls to HOOD algorithms allows the authors to derive convergence rates for the proposed reductions. Combining these results with known methods for strongly convex optimization, the authors derive algorithms with optimal convergence rates in important optimization model classes. The results are illustrated on Lasso and SVM problems with benchmark datasets.
Overall, the paper is convincing and well written. The problem is very relevant for machine learning applications and the proof arguments are quite convincing. I could not find major flaw in the paper.  I have several concerns to raise regarding this work. - Presentation and comparison with existing methods of similar flavor is not really fair. Smoothing has a long history in optimization and recent work on accelerated methods through strong convexity are not properly presented. Some elements of discussion are missing and no concurrent algorithm is implemented in the numerical section. - The authors put a lot of emphasis on what they call the "old" or "classic" reduction methods. There is no reference about these methods and related complexity results mentioned in the paper. Furthermore, most of the discussion about comparison with state of the art focuses on these "old" reductions. I am not really convinced that this is a fair baseline. - Some aspects of the discussion regarding lasso are very inaccurate.  %% Details about the main comment  Comparison with existing literature:  Smoothing nonsmooth objectives is a relevant topic in optimization literature. For example reference [20] in the main text or the following reference Beck, Amir, and Marc Teboulle. "Smoothing and first order methods: A unified framework." SIAM Journal on Optimization 22.2 (2012): 557-580. I guess the superiority of the proposed framework is the "anytime" property, but this deserved to be presented more clearly.  The authors present the work in [8] and [15] as being identical to the "traditional regularization reduction". I do not understand this point. These works are based on extensions of the proximal point algorithm which indeed adds a strongly convex term with a fixed modulus of strong convexity at each step. But the term itself depends on the iterate counter which is very different from what is described as the classical reduction method. These methods may suffer from log factors, but I disagree with the fact that they suffer from bias. There is no discussion about parameters that are required for the proposed algorithm compared to these methods. For example, in Theorem 3.1, AdaptReg requires an estimate of the distance to the optimum point. How to estimate this? Is the method robust to under estimation of this parameter? Do concurrent methods require this parameter. To my understanding, Catalyst only requires a bound on suboptimality which is easier to estimate. Similar comments hold for Theorem C.2. Finally, no concurrent or even baseline method is implemented apart from the "old" reduction which I think weakens considerably the numerical illustration. Indeed, they mostly illustrate the fact that the "old" reduction is not an anytime method.  The complexity results regarding the old reduction method in section 1.1 should be justified by proper references.   %% Additional questions and suggestions  The bibliography should be in alphabetical order. Ref [27] is not cited in the main text.  On line 24 it is argued that "for a non-SC objective such as Lasso, its dual is not even be well-defined". First, there is a typo and, second, this does not make sense to me. The dual is well defined, feasible and its optimal value is attained.  On page 3, the authors mention optimality of the convergence rates they obtain and justify them by citing reference [1]. From my understanding, the mentioned paper does not focus on lower bounds and there should be a better reference.  In remark 3.2, the authors mention that it is much easier to tune \sigma_0 than \sigma because this is "seen from the plots". This remark is repeated in the numerical section and I do not find that this discussion is convincing.   The presentation of Section 6.1 is very fuzzy. Contrary to what is said, the lasso problem is not smooth. There must be a confusion somewhere. What is meant by indirect solver on line 246?  The whole analysis is based on the HOOD property and the numerical illustration is done with methods which do not satisfy this property. This is a bit disappointing.  Since the duality gap is only an upper bound on suboptimality, the remark on line 255 is only an heuristic and is not sufficient to ensure that the results of the paper apply. This could be precised. A similar comment holds for line 283.  The numerical section does not describe what happens for a bad choice of \sigma_0.  At the end of appendix A, it is mentioned that it is not possible to compute duality gaps for the Lasso. I know at least one possibility from the Appendix of Julien Mairal's Thesis. Moreover, the norm of the gradient is not really the best indicator of suboptimality since it is not 0 in general at the optimum for the lasso. May be the authors refer to the gradient mapping (which includes the prox step). Or may be this is related to the confusion that I pointed out earlier.


SDP Relaxation with Randomized Rounding for Energy Disaggregation
We develop a scalable, computationally efficient method for the task of energy disaggregation for home appliance monitoring. In this problem the goal is to estimate the energy consumption of each appliance over time based on the total energy-consumption signal of a household. The current state of the art is to model the problem as inference in factorial HMMs, and use quadratic programming to find an approximate solution to the resulting quadratic integer program. Here we take a more principled approach, better suited to integer programming problems, and find an approximate optimum by combining convex semidefinite relaxations randomized rounding, as well as a scalable ADMM method that exploits the special structure of the resulting semidefinite program. Simulation results both in synthetic and real-world datasets demonstrate the superiority of our method.

This paper proposed method to find better approximate solution for energy disaggregation or non-intrusive load monitoring (NILM) problem based on the previous work done by Kolter & Jaakkola (2012) who used an additive factorial HMM to model the energy consumption. The authors combined a SDP relaxation and randomized rounding (Park & Boyd, 2015) and applied ADMM (Boyd, 2010). The proposed method achieved superior performance to the existing method.
It is meaningful that the proposed method perform outstandingly well on simulation results in aspects of accuracy. However, this paper lacks academic novelty and originality in that the proposed method solely combined existing works. The authors argue that proposed algorithm is scalable and computationally efficient, but there is no explanation about computational complexity on method or experimental results. It is necessary to show effectiveness in aspects of computational complexity. Alternatively, comparison with existing methods on execution time or experiments using large-scale data could be provided. This paper is well written overall and states essential information. The author should modify some mistakes in expression including the title of section 6 and its subsection, line 75 and line 223. It would be better to represent important figures or tables on the manuscript not on the supplementary material in spite of the limited space. In addition, conclusion part is too short.


Fast Algorithms for Robust PCA via Gradient Descent
We consider the problem of Robust PCA in the fully and partially observed settings. Without corruptions, this is the well-known matrix completion problem. From a statistical standpoint this problem has been recently well-studied, and conditions on when recovery is possible (how many observations do we need, how many corruptions can we tolerate) via polynomial-time algorithms is by now understood. This paper presents and analyzes a non-convex optimization approach that greatly reduces the computational complexity of the above problems, compared to the best available algorithms. In particular, in the fully observed case, with r denoting rank and d dimension, we reduce the complexity from O(r 2 d 2 log(1/ε)) to O(rd 2 log(1/ε)) -a big savings when the rank is big. For the partially observed case, we show the complexity of our algorithm is no more than O(r 4 d log d log(1/ε)). Not only is this the best-known run-time for a provable algorithm under partial observation, but in the setting where r is small compared to d, it also allows for near-linear-in-d run-time that can be exploited in the fully-observed case as well, by simply running our algorithm on a subset of the observations.

This paper presents an efficient algorithm for the problem of robust PCA. The key idea is estimating a good initial solution followed by gradient descent for improvement. Compared to a recent (non-convex) robust PCA solver, the computational complexity is reduced by a factor of "r", where "r" denotes the rank of the true matrix. The algorithm is straightforward, easy to follow. The main technical contribution is the linear convergence guarantee which appears novel.
This is an interesting work that presents the perhaps first theoretical guarantee for a widely used optimization technique of the problem of robust PCA. I think it has potential and should be somewhere between poster and oral. During the rebuttal period, I would suggest the authors to address the following minor concerns.  - It appears to me that the algorithms (Algorithm 1 and Algorithm 2) is easy to follow. That been said, it is not clear how one is able to know the true rank "r" and the corruption fraction "alpha". The projection step (Step 7 and Step 8) requires a knowledge of the incoherence parameter. How to estimate it? If one only knows some upper bound on the rank and corruption fraction (which should be the practical case), is it possible to show that all the theoretical results still hold?  - Theorem 1 states that the initial deviation for "(U_0, V_0)" is upper bounded by a quantity which is proportional to "sqrt(r/kappa)". This means when "r" is large, the initial estimate is not good enough. Given this, why does the algorithm still succeed by projecting the iterates onto the space of "mathcal{U}" and "mathcal{V}" which are determined by "U_0" and "V_0", respectively? Please give more intuition on it.  - The sample complexity is not optimal here. The authors claim that it can be improved by utilizing the output of [21]. For the sake of concreteness, the authors should frame such an argument more rigorously, e.g., "if we apply the AltProj[21] for initialization, our Theorem 1 can be strengthened to d(U_0, V_0; U^*, V^*) < ... In this way, we obtain an improved robustness result ..." Otherwise, it is tricky to check the correctness of Remark 2.  - When analyzing the computational complexity and robustness, the authors presume that "kappa" is a constant and compare the theoretical result to state-of-the-art solvers. This is somewhat unfair since "kappa" might scale with the dimension. A good practice is carefully discuss when the presented algorithm dominates previous solvers and when it does not, e.g., "when kappa = O(1), we observe that our algorithm is an order of magnitude faster ... when kappa = O(d), we find that ...". It would be better to provide several examples where "kappa"  behaves like a constant.   --------------------------------- Updates:  The author feedback is convincing. It is a nice work to me.  A minor comment: It seems that another parallel work [A] also considers online alternating minimization. It would be good if the authors could comment on the difference.  [A] Provable Efficient Online Matrix Completion via Non-convex Stochastic Gradient Descent, arXiv:1605.08370


Interpretable Distribution Features with Maximum Testing Power
Two semimetrics on probability distributions are proposed, given as the sum of differences of expectations of analytic functions evaluated at spatial or frequency locations (i.e, features). The features are chosen so as to maximize the distinguishability of the distributions, by optimizing a lower bound on test power for a statistical test using these features. The result is a parsimonious and interpretable indication of how and where two distributions differ locally. We show that the empirical estimate of the test power criterion converges with increasing sample size, ensuring the quality of the returned features. In real-world benchmarks on highdimensional text and image data, linear-time tests using the proposed semimetrics achieve comparable performance to the state-of-the-art quadratic-time maximum mean discrepancy test, while returning human-interpretable features that explain the test results.

The paper considers the classical two sample problem where one wants, from the observation of two independent samples in R^d, X_1^n and Y_1^n with respective distribution P and Q, be able to test whether P = Q or P \neq Q. The solution investigated in the paper, which is due to Chwialkowski et al., is very interesting. It is based on a kernel k : (R^d)^2 → R and a set of J test locations (v_j)^J_{j=1}. For any i = 1,...,n, one builds the vectors z_i with coordinates k(x_i,v_j) − k(y_i,v_j), j = 1,...,J and measure the discrepancy between P and Q with the T-statistic \hat{\lambda}_n=n\bar{z}_n(S_n+\gamma_nI)^{-1}\bar{z}_n, where \bar{z}_n=\frac1n\sum_{i=1}^nz_i, S_n=\frac1{n-1}\sum_{i=1}^n(z_i-\bar{z}_n)(z_i-\bar{z}_n)^T. The authors Chwialkowski et al. proved in particular that the asymptotic distribution of \hat{\lambda}_n, when n → ∞ and the other parameters are fixed, is \chi^2(J). The present paper also consider an associated test, called SCF-test which relies on the same ideas. The main question is how should we choose the test locations (vj)_{j=1}^J to optimize the performance of this test (the choice of the kernel k or of some scaling parameters is also relevant). This is precisely the question considered by the author, who proceed as follows. First, they prove a lower bound on the power of the test that is increasing with the limit \lambda = \mu^T \Sigma^{−1}\mu of \hat{\lambda}_n/n as n → ∞. Then, as \lambda is unknown, they cut the sample in two halves, the first one is used to estimate the parameters \mu and \Sigma and therefore estimate the function \lambda of the parameters of interest (v_j)_{j=1}^J. The second is used to select the parameters maximizing this estimator of \lambda. The author proves the consistency of the estimator of \lambda and test its procedure on both toy Gaussian regression models and in two real data-sets in a simulation study concluding the paper.
I found the approach of the problem very interesting and the solution quite elegant and, even if I did not check the details of all proofs, they seem correct. This is why I would like to see the paper published. However, and this is my main concern, I really think that the author should make a strong effort of presentation of their results. In the present state, I think that it is quite hard to read due to the lack of many details, not completely honest on some aspects and both theorems are unnecessarily way too technical. I will now detail some of these points.  You should improve the presentation of the paper. Some important formulas, like the definition of your main quantities (for example zi p3 l98) should be isolated in an equation to ease the reading. Moreover, please define the quantities in the text when you use them in your main results, for example the Frobenius norm used p3 l121 in your first result is only defined in Section C1 of the supplementary material. These are really not details, as I said it makes the paper very hard to understand. To improve the presentation, I also suggest that you avoid the sketchs of proofs that I found quite obscur and not really informative. The same holds for the comment after Proposition 1 giving another intuition that maximizing \lambda would lead to better performances, as you said, Proposition 1 does it and better. Also, p4 at the beginning of the Section Contribution, you briefly change from a parameter k to designate the kernel to \sigma before going back to k and use \sigma as a tuning parameter.  You refer several times to the fact that your procedure is adapted to the large dimension setting. You should be more precise here. I believed that it meant that d or J can be large compared to the number n of observations. However, it looks like your test has only confidence level \alpha in an asymptotic framework where n → ∞ while the other parameters remain fixed? Also, in your comment after Theorem 2, you get the rate O(n^{−1/4}) only if the VC dimension of your class of kernels is not allowed to grow with n. Finally, in your simulations study, I think that, in all toy models, n is larger than d and much larger than J, and, for example on Fig 3, it is not clear that your procedure remain effective when d gets closer to n.  You should finally improve the presentation of your theorems. For example, since you introduce some ”constants” in Proposition 1 depending on B, J to express L(\lambda_n), why do these parameters and meaningless absolute constants still appear in the expression of L(\lambda_n)? You should really write a more synthetic and comprehensive result in the main text, and probably keep this result for the supplementary material. The same comments hold for Theorem 2.


Sorting out typicality with the inverse moment matrix SOS polynomial
We study a surprising phenomenon related to the representation of a cloud of data points using polynomials. We start with the previously unnoticed empirical observation that, given a collection (a cloud) of data points, the sublevel sets of a certain distinguished polynomial capture the shape of the cloud very accurately. This distinguished polynomial is a sum-of-squares (SOS) derived in a simple manner from the inverse of the empirical moment matrix. In fact, this SOS polynomial is directly related to orthogonal polynomials and the Christoffel function. This allows to generalize and interpret extremality properties of orthogonal polynomials and to provide a mathematical rationale for the observed phenomenon. Among diverse potential applications, we illustrate the relevance of our results on a network intrusion detection task for which we obtain performances similar to existing dedicated methods reported in the literature.

This paper uses sum-of-squared (SOS) polynomials (i.e. the set of polynomials of degree 2d which can be written as the square of polynomials of degree d) to detect outliers in point sets. In contrast to kernel-based methods such as Gaussian processes, the evaluation of an SOS polynomials depends on the number of monomials (and thus on the ambient dimension) rather than on the number of observations. Thus the approach seems obviously advantageous for evaluating typicality in very large point sets of relatively low dimensionality. The paper provides some nice theoretical analysis that helps to explain empirically-observed properties of SOS polynomials that make them useful for typicality. Finally, the authors apply the technique to a dataset of network
This paper is clear and seems thorough and technically sound, at least up to the depth at which I could follow it. If it is indeed the case that the presented empirical observation of the utility of SOS polynomials for describing typicality in point sets has not been previously noted in the literature, then it would be a clearly useful contribution. The experimental results show that the SOS polynomial technique is competitive with other techniques hand-designed to detect typicality while also being simple and well-justified from a theoretical standpoint.  My only concern is regarding the fitness of the presented work for presentation in NIPS. It seems that this approach would be extremely useful for geometric settings in which the dimensionality is low, e.g. 2 or 3, but the number of points can be in the millions or more, perhaps for 3D point-cloud de-noising applications. As for machine learning, the paper presents one example that seems reasonable, but it would be useful if some additional examples of where this approach my prove advantageous were given, or even just some additional motivation in the text describing potential applications.  Minor points on clarity:  I assume x bar in section 4.4 (introduced in line 239) is referring as usual to the mean of a set of points in R^p but that wasn't made entirely clear and I don't think it was used previously.  On line 257 in section 5 it says that the network intrusion dataset consists of labelled vectors in R^3, but in lines 441-442 of Appendix C it says "the four most important features" are used. Is this a typo or is there some sort of projection from the four features to a 3D space being used?  A very minor point, but it seems that the section describing affine invariance is really describing affine co-variance, i.e. if you transform the point set using an affine transformation, the resulting SOS is different (thus not invariant) but it is equivalent to transforming the underlying coordinate system of the polynomial (thus co-variant).


Learning Sparse Gaussian Graphical Models with Overlapping Blocks
We present a novel framework, called GRAB (GRaphical models with overlApping Blocks), to capture densely connected components in a network estimate. GRAB takes as input a data matrix of p variables and n samples and jointly learns both a network of the p variables and densely connected groups of variables (called 'blocks'). GRAB has four major novelties as compared to existing network estimation methods: 1) It does not require blocks to be given a priori. 2) Blocks can overlap. 3) It can jointly learn a network structure and overlapping blocks. 4) It solves a joint optimization problem with the block coordinate descent method that is convex in each step. We show that GRAB reveals the underlying network structure substantially better than four state-of-the-art competitors on synthetic data. When applied to cancer gene expression data, GRAB outperforms its competitors in revealing known functional gene sets and potentially novel cancer driver genes.

This paper presents a novel method GRAB that jointly learns the network structure and assignment of variables into potentially overlapping blocks. The authors also proposed a reliable optimization algorithm with the block coordinate descent method. Although presented in the context of Gaussian graphical models, GRAB can be generalized to learning of other types of networks. The proposed method is also illustrated in simulations and a real data application to demonstrate its usefulness in uncovering meaningful block structures.   Overall it is an interesting and important work to integrate network estimation and community detection into one unified problem. The optimization is done seamlessly. However, there are also some issues to be addressed, mainly regarding the identifiability of the block memberships.
 (1) In this framework, the entries in Z are within [-1, 1]. In the context of stochastic block model, there is also an assignment matrix Z with entries {0, 1} (see Rohe et al 2011 Spectral clustering and the high-dimensional stochastic blockmodel). How does Z here (in the special binary case) compare with the Z matrix in Rohe et al. 2011?  (2) In section 2.2, could you explain how the constraint |Z|_F \le beta prevents all regularization parameters from becoming zero?  (3) In section 2.2, the constraint (c) |Z|_2 < tau is to prevent the case where all variables are assigned to one block. Why is it not allowed for all variables to be assigned to one block? It's possible that one wants to analyze the interaction network among genes from the same pathway (the same block). In the case of a chain graph, how many blocks are there?    (4) In section 4.2, the authors explain how to obtain blocks from the estimated Z matrix. The assignment of the block membership seems rather exploratory. In view of the problem formulation in (3) with the penalty matrix is lambda * (1 - (ZZ^T)_{ij}), there might be identifiability issue. That is, lambda_1 * (11^T - Z_1 Z_1^T) = lambda_2 * (11^T - Z_2 Z_2^T) for (lambda_1, Z_1) \ne (lambda_2, Z_2). Consider the example of a chain graph. In this case, how should one interpret the matrix Z and the following assignment of block memberships?     (5) In Section 3.3, equation (8): diag((W)) --> diag(W). Similar corrections should be made one line above equation (9). When introducing equation (11), lambda_i's are used as eigenvalues, whereas lambda was used previously as the penalty parameter. Maybe a different notation is better. In equation (13), the rightmost parenthesis is missing. Further, do you still need the constraint diag(W) \le 1 in equation (13)?   (6) In Section 4.2, some details on how the results are obtained are missing. For example, why and how are the between-block edges removed from further analysis? If this is the case, should similar procedure be done for the competing method? In particular, the comparison in Figure 5 seems to be unfair for the Glasso. It is true that GRAB directly returns the block assignment, however one may also recover the blocks based on the estimated adjacency matrix from Glasso. Have you tried building blocks for the Glasso? In that case, it might be fair to compare the estimated blocks for both methods.


Variational Inference in Mixed Probabilistic Submodular Models
We consider the problem of variational inference in probabilistic models with both log-submodular and log-supermodular higher-order potentials. These models can represent arbitrary distributions over binary variables, and thus generalize the commonly used pairwise Markov random fields and models with log-supermodular potentials only, for which efficient approximate inference algorithms are known. While inference in the considered models is #P-hard in general, we present efficient approximate algorithms exploiting recent advances in the field of discrete optimization. We demonstrate the effectiveness of our approach in a large set of experiments, where our model allows reasoning about preferences over sets of items with complements and substitutes.

The authors consider the problem of inference in mixed submodular models, where the probability of a set "A" is proportional to exp(F(A) + G(A)), where F and G are submodular (i.e. -G is supermodular). Previous work was only able to handle a single submodular or supermodular function in the exponent.  They then proceed to upper bound the partition function of the model by using a modular function that upper bounds F(A) - G(A). It is easy to calculate the partition function of a modular function. They then optimize over upper bounds to find an estimate of the partition function that is reasonably tight. Their optimization procedure involves an inner bound on the set of possible modular functions, then a block coordinate descent procedure using the Frank-Wolfe algorithm. The authors then empirically show that their approach achieves reasonable results in a product recommendation experiment.
The authors consider a useful generalization to inference in submodular models, which has been receiving a good amount of attention in recent conferences.  Their paper is well written and synthesizes many results in the literature and can provide an example of how to use previously developed tools.  However, the approach they take is not entirely novel compared to past work. With limited new theoretical results, to make the work a solid contribution it should have a more in-depth experimental analysis. Their experiment does not sufficiently compare to approaches outside the variational inference for submodular model framework. Specifically, they should compare to other approaches for estimating marginals, such as MCMC, and other approaches for estimation (e.g. logistic regression, SVM, boosting etc.) or mention why other approaches would not work.  == post-rebuttal answer==  Thank you for your rebuttal. After reading the other reviews I would agree that this is somewhat incremental, with many of the core ideas coming from previous work. However I found the problem to be interesting and believe that there are some nice algorithmic and modeling contributions. I would have liked to see a more extensive experimentation section with details on runtimes and other accuracy baselines to better judge the impact of this approach, especially since the algorithms are more heuristical in nature.


New Liftable Classes for First-Order Probabilistic Inference
Statistical relational models provide compact encodings of probabilistic dependencies in relational domains, but result in highly intractable graphical models. The goal of lifted inference is to carry out probabilistic inference without needing to reason about each individual separately, by instead treating exchangeable, undistinguished objects as a whole. In this paper, we study the domain recursion inference rule, which, despite its central role in early theoretical results on domain-lifted inference, has later been believed redundant. We show that this rule is more powerful than expected, and in fact significantly extends the range of models for which lifted inference runs in time polynomial in the number of individuals in the domain. This includes an open problem called S4, the symmetric transitivity model, and a first-order logic encoding of the birthday paradox. We further identify new classes S 2 FO 2 and S 2 RU of domain-liftable theories, which respectively subsume FO 2 and recursively unary theories, the largest classes of domain-liftable theories known so far, and show that using domain recursion can achieve exponential speedup even in theories that cannot fully be lifted with the existing set of inference rules.

The authors return to "domain-recursion" as a technique for lifted inference (that is, inference, focused on models that combine probabilities and logical construction, that avoids grounding the whole model). The technique is shown to tackle problems that have defied liftable inference so far. Also the authors characterise a new class of liftable models. Small scale tests show the gains from the technique (somewhat unsurprisingly, as we know that liftable inference, whenever applicable, may lead to exponential gains).
The authors present a novel assessment of domain-recursion, and show that it can handle previously open problems. This is very interesting, and the paper is clear in its main goals and results. Overall the paper is an incremental effort in revisiting previous results and previous algorithms; the contribution is focused and narrow, but it can be quite valuable for other researchers interested in this topic. I should note that this is a difficult topic. The paper requires some polishing but still it contains a contribution.  Concerning the presentation, I find that the authors must add some comments on their assumptions. For instance, what exactly is the language they allow (I know that previous papers on this subject follow the same path, but still this is a something that must be fixed). I mean, is a formula an expression without quantifiers, and understood as universally quantified? But then what is a "theory in first-order logic" mentioned in page 2 (is it with quantifiers)? And then why is it that the birthday paradox uses an existential quantifier (and then the authors mention that it can be dealt with by Skolemization, but how? The exact way Skolemization is used should be shown). I find that the description of the language in Section 2 should be clear and leave no space for doubts. The authors must improve that aspect of the paper. Later, the same effort should be made to clarify FO^2 and its variants (do they accept quantifiers? where? how?).  Note for instance that if formulas are actually open formulas without quantifiers, then it is hard to imagine how they are interpreted for weighted first-order model count; I guess such a formula must be closed (but then, I suppose all formulas are implicitly closed with universal quantifiers?).   A relatively small point: I find Section 2.1 not to be very useful, the example with MLN is interesting but not needed here. Perhaps this can be shortened so that space is left for more important material.  A point about notation: it is funny that a PRV, say Q, appears as q in formula, while a logical variable, say x, appear capitalized as X; what is the logic concerning capitalization here?  There seems to be something wrong with the expression in the section about Shattering; the first expression has two quantifiers, but it should only have one (??).  It is a pity that S4-clauses and symmetric transitivities are not shown, however shortly, to be liftable. They are not in FO^2+, so the reader is left without any clue as to why they are liftable. Perhaps some space can be found to add some explanation on this matter.  I find the definition of FO^2+ to be harder to parse than necessary; please improve Definition 1.   One point I did not quite get: can one know the gains that will be obtained from domain-recursion BEFORE applying it? I mean, suppose we have a model and we cannot solve by other rules. We check and see that the model is NOT in FO^2+. Can we know whether domain-recursion will be useful, or not?   At the end of Section 6, the authors mention that costs are n^5 and n^2. How was that concluded? Just by empirical testing indicated in the graphs? But the gains should be derived from the algorithms that are used; if it is liftable against something exponential, the gains are exponential. I did not get the meaning of n^5 and n^2 there.  Concerning the text, here are some specific comments: - Page 1, line -4: the concept of domain-liftability may seem similar to data complexity, but actually lqe-liftability as defined by Van den Broeck and coauthors is more akin to data complexity; I don't think domain complexity has such a clear counterpart in the database world. - Is footnote 1 correct (perhaps conjoin X != Y)? - Right before Section 2.3: "process exponential in the number of individuals" is quite imprecise; this depends on the algorithm used to handle grounding, I suppose.  - Right before Section 6, the authors use "suggesting", but I suppose they mean "showing". - Footnote 3 misses an ending period. - Page 8: "straightforward" instead of "straight-forward". - Problems with capitalization in references: prolog, markov.


MoCap-guided Data Augmentation for 3D Pose Estimation in the Wild
This paper addresses the problem of 3D human pose estimation in the wild. A significant challenge is the lack of training data, i.e., 2D images of humans annotated with 3D poses. Such data is necessary to train state-of-the-art CNN architectures. Here, we propose a solution to generate a large set of photorealistic synthetic images of humans with 3D pose annotations. We introduce an image-based synthesis engine that artificially augments a dataset of real images with 2D human pose annotations using 3D Motion Capture (MoCap) data. Given a candidate 3D pose our algorithm selects for each joint an image whose 2D pose locally matches the projected 3D pose. The selected images are then combined to generate a new synthetic image by stitching local image patches in a kinematically constrained manner. The resulting images are used to train an end-to-end CNN for full-body 3D pose estimation. We cluster the training data into a large number of pose classes and tackle pose estimation as a K-way classification problem. Such an approach is viable only with large training sets such as ours. Our method outperforms the state of the art in terms of 3D pose estimation in controlled environments (Human3.6M) and shows promising results for in-the-wild images (LSP). This demonstrates that CNNs trained on artificial images generalize well to real images.

Authors propose a method to generate training samples for the task of 3D pose estimation from single monocular images. To this end they utilize 2 data sources: 3D motion capture data and images annotated with 2D poses. For a given 3D pose and a camera viewpoint they retrieve N images (where N is number of joints), which have the most similar local geometry around respective joints as the 2D projection of the source 3D pose. Then, they stitch the N images into a coherent one with pose aware smoothing. The resulting synthetic dataset is added to the existing dataset which is annotated with 3D poses. 3D poses are clustered into K=5000 pose classes and a holistic CNN is trained for a classification task. At test time 3D pose is estimated as average pose of top scoring classes. This method achieves state-of-the-art performance on Human3.6M dataset.
- Technical quality.  Authors compare their method to a very recent state-of-the-art method and use the same evaluation protocol. They present detailed evaluation of how synthetically generated data fairs against real data and how using them both together improves performance. Additionally authors demonstrate the advantage of formulating 3D pose estimation as a classification problem over direct regression to joint coordinates. This is facilitated by a large number of training samples, which they show is crucial for the classification CNN: in the small data regime full body regression actually outperforms 3D pose classification.  However, authors only evaluate their method using evaluation protocol described in [17, 40]. It would be nice to show how method compares to [18, 19] which use a different protocol on Human3.6M, even though it is an easier problem according to [17].  - Novelty/Originality  To the best of my knowledge this work is the first to synthesize training samples from real world images for the task of human pose estimation. This is an alternative approach to rendering images given the 3D pose of a subject. Despite the fact that generated images do not match real-world images in realism, using them is beneficial.  - Potential impact or usefulness  In order to train Deep Neural Networks large datasets are required, and this work explores how to generate such training sets. Therefore it can potentially be interesting to the community.  Since the method allows to generate diverse samples for 2D pose estimation as well, showing results on 2D pose estimation would demonstrate whether the proposed approach is applicable to other problems. Thus an experiment with augmenting typical 2D pose training sets such as MPII that shows an improvement in 2D pose estimation would be of great value. Otherwise, quantitative evaluation on only one dataset slightly questions the generality of the approach.  The evaluation on LSP while interesting, does not give insight of how the proposed data augmentation method can push the 2D pose estimation, because it does not use an established evaluation metric.  - Clarity and presentation  The data generation method is described with great detail and I had no problem understanding it. The paper reads well overall, however there are a number of spelling mistakes to be corrected.  - Final assessment  Overall, I like the paper. Authors clearly demonstrate that adding training samples, generated with their method, improves performance of 3D pose estimation  using a classification-based CNN. However I feel that quantitative evaluation on just one benchmark is not enough to conclude on generality of the data generation method, especially because generated images do not appear entirely realistic.


Neurons Equipped with Intrinsic Plasticity Learn Stimulus Intensity Statistics
Experience constantly shapes neural circuits through a variety of plasticity mechanisms. While the functional roles of some plasticity mechanisms are wellunderstood, it remains unclear how changes in neural excitability contribute to learning. Here, we develop a normative interpretation of intrinsic plasticity (IP) as a key component of unsupervised learning. We introduce a novel generative mixture model that accounts for the class-specific statistics of stimulus intensities, and we derive a neural circuit that learns the input classes and their intensities. We will analytically show that inference and learning for our generative model can be achieved by a neural circuit with intensity-sensitive neurons equipped with a specific form of IP. Numerical experiments verify our analytical derivations and show robust behavior for artificial and natural stimuli. Our results link IP to nontrivial input statistics, in particular the statistics of stimulus intensities for classes to which a neuron is sensitive. More generally, our work paves the way toward new classification algorithms that are robust to intensity variations.

Neurons with Intrinsic Plasticity Learn Stimulus Statistics  In the paper, the author(s) build a generative product-poisson-gamma model to recognize images with wide dynamic range, by extending a previously used Poisson mixture model. An equivalent neural circuit model with unsupervised learning was also proposed to implement EM-like algorithms to find the model parameters in generative model. In the learning  rule, the synaptic weights are changed according to a Hebb like rule with synaptic scaling, and intrinsic parameters lambda (Eq.7 in the paper) are updated depending on  the net input. It is very interesting that the fixed points of the network updating parameters can have the same form with EM algorithm. However, I have the following  questions about this paper:       1. In the paper, the author claims that the parameter lambda corresponds to a neuron's intrinsic parameter, and updating of lambda implies to plasticity of intrinsic parameters. For me, I think here lambda is like a input gain control parameter in the neuron model, as it controls the input current that leads to saturation of the neuron.   This is a very abstract intrinsic parameter, is it suitable to name it as intrinsic plasticity?  2. The number of neurons configured in the hidden layer. For me, I think in the model, if the number of hidden neuron equals to the classes in input data, then the  network may have a good classification ability. While this is not always the case in reality, what would happen to a network if the number of hidden neuron does not  equal to the number of classes in input data? Will the classification performance would degrade?  3. Time scale. It can be seen that, the synaptic weights and intrinsic parameters are updated at the same time, that is, the updating time  constant would be the same. However, contrast of natural images and input pattern would not likely change with a same time constant in reality.  So, what would happen in your model if synaptic weights and intrinsic parameter are updated with different time scale?
This is a theoretical beautiful work. It builds a neural circuit model with unsupervised learning to get an theoretical optimal fixed point, which is highly corresponded to EM learning algorithm result. This work considers the effect of contrast variation by extending a previously Poisson mixture model. The main concern is, what would happen when the hidden neuron number does not  equals to the pattern class number, e.g., the number of hidden neuron is larger than the pattern classes.


Asynchronous Parallel Greedy Coordinate Descent
In this paper, we propose and study an Asynchronous parallel Greedy Coordinate Descent (Asy-GCD) algorithm for minimizing a smooth function with bounded constraints. At each iteration, workers asynchronously conduct greedy coordinate descent updates on a block of variables. In the first part of the paper, we analyze the theoretical behavior of Asy-GCD and prove a linear convergence rate. In the second part, we develop an efficient kernel SVM solver based on Asy-GCD in the shared memory multi-core setting. Since our algorithm is fully asynchronous-each core does not need to idle and wait for the other cores-the resulting algorithm enjoys good speedup and outperforms existing multi-core kernel SVM solvers including asynchronous stochastic coordinate descent and multi-core LIBSVM.

This paper considers asynchronous parallel greedy coordinate descent (Asy-GCD) algorithm for minimizing a smooth function with decomposed constraints.  The algorithm works in a parameter server framework, with each worker performing greedy CD asynchronous.  This is different from [17] which picks the coordinate stochastically.  Efficient coordinate search algorithm is proposed for kernel SVM by maintaining the gradient, which is updated by atomic writes.  The results are mainly two folds.  First in theory, a linear convergence rate is established.  Second in practice, by using kernel caching and sensible variable partitioning, the proposed algorithm outperforms existing solvers in a shared-memory multi-core setting.
This paper is well written and the innovations and improvements can be seen clearly.  Although changing stochastic into greedy is not that big innovation and the O(l) algorithm for coordinate selection based on maintaining the gradient is also not new, I can kind of appreciate the practical improvement after putting together some existing tricks (Sec 4.3).  My major concern is, however, the rates of convergence.  There are five “constants” involved in Theorem 2, \kappa, L_res, L_max, L, and L_T.  It is critical to unfold their values for concrete problems, e.g. the kernel SVM problem considered in the paper.  Linear rates are useful only if its condition number depends on the dimensions of the problem mildly.  What is most unsettling is the L_T^2 term in b.  How does L_T depend on T?  If it is \Omega(\sqrt{T}), then this will translate to an iteration complexity that is linear in T.  Since “T is usually proportional to the total number of cores involved in the computation”, this will cancel the speedup proffered by parallelization and asynchronous updates.  It is therefore crucial that the paper clarify the value of these “constants” explicitly.  =================== Post-rebuttal:  I read the rebuttal, and now feel less worried about the dependence on L_T (or T). However, based on the numbers provided in the rebuttal, now I am rather worried about the constants \kappa and L.  Using the rebuttal's argument that b is almost a universal constant, the rate of convergence is roughly  L n \kappa^2 \log (1/\epsilon)  The values of L and \kappa provided in the rebuttal seem to grow linearly in n.  This is unsettling because such a high order dependency on n essentially leaves the linear rate in vain.  The submission allows parallelization up to n^{1/4} cores.  So roughly speaking, the cost is   O(n^{15/4} \log (1/\epsilon))   15/4 might be over-estimated a little, but it is far worse than the following rate for training nonlinear kernel SVM given by [a]:  O(n / \epsilon).  The disadvantage is even more evident considering that \epsilon rarely needs to be less than 1/n (consider n = 4*10^5 in the experiment where #example is denoted as \ell instead of n).  Having said that, I think think this paper actually has some high quality material.  The experiments are surely good.  The linear rates were first shown for this problem, although further refinement is needed on its condition number.  Overall, this paper is above the acceptance threshold of NIPS, and I've updated my ratings accordingly.  [a] Bernd Gärtner and Martin Jaggi, Coresets for Polytope Distance, Symposium on Computational Geometry, 2009.