Parallelizing MCMC with Random Partition Trees
The modern scale of data has brought new challenges to Bayesian inference. In particular, conventional MCMC algorithms are computationally very expensive for large data sets. A promising approach to solve this problem is embarrassingly parallel MCMC (EP-MCMC), which first partitions the data into multiple subsets and runs independent sampling algorithms on each subset. The subset posterior draws are then aggregated via some combining rules to obtain the final approximation. Existing EP-MCMC algorithms are limited by approximation accuracy and difficulty in resampling. In this article, we propose a new EP-MCMC algorithm PART that solves these problems. The new algorithm applies random partition trees to combine the subset posterior draws, which is distribution-free, easy to resample from and can adapt to multiple scales. We provide theoretical justification and extensive experiments illustrating empirical performance.

Acknowledge the author's rebuttal.I overall maintain my general sentiment about the paper.--In this paper the authors describe and evaluate a method to aggregate posterior samples from an MCMC algorithm run over partitions of a dataset, where the observations are conditionally independent given the model parameters being estimated.One of the primary limitations of existing work that the authors intend to address is that existing aggregation methods poorly handle distributions which are multimodal or have skew (i.e., those which are not Gaussian).Their method to do this is by generating histograms of the parameter space (where the histograms are shared by partitions/samplers), then counting samples within the blocks of these histograms and aggregating across partitions by simply summing observations.The authors then illustrate their method with several experiments - some toy problems, Bayesian logistic regression on a synthetic dataset, and some larger data sets - and illustrate that their method leads to consistently better estimates of the posterior.What I concerns me about this paper is that I feel that the complexity of their method should be justified with more demonstration that their posterior estimate is actually better in practice than with alternative methods, especially on large datasets, for which the posterior very possibly is near-normal (more on this below).What I like about it is that, at least as a first cut, it uses a method which is simple in theory (basically, histograms and counting) to aggregate posterior samples from disjoint partitions into a full posterior estimate.What I think can be improved is that the complexity of the method should be better justified, and the explanation of it should be cleaned up (because while it is simple in theory, it appears complicated in practice to the extent that it is not clear that it is worth the added complexity).More concretely, my primary suggestion for this paper to the authors is to devote more effort to describing the partition algorithms that they propose.I believe they could do this by discussing related work a bit less (focusing mainly on the PDE equation) and by moving the remaining proofs to the appendix (but still stating the consistency results).By doing this, the authors could make it easier for readers to implement and tune their own implementations of this algorithm.As it is, I wouldn't want to be the grad student tasked with implementing their algorithm.It would be worth seeing some discussion about when the assumption of skewness in a posterior with many observations (and moderate dimensionality) actually is a problem when aggregating posteriors.Particularly, the "Bayesian central limit theorem" indicates that the posterior approaches normality when there are many samples and moderate dimensionality: http://www.stat.cmu.edu/~larry/=stat705/Lecture14.pdf.This large-observation scenario is the exact scenario this paper is aimed to address.When each subset has many observations, the resulting first and seconds moments may actually capture the distribution quite well, so the complexity of this method may not be justified.In addition, I have several more comments: - It would be good to see more discussion about previous methods for creating partition trees and how the authors' methods relate to these. - More motivation should be given in the introduction to the motivating problems; the algorithm is jumped into without enough of a clear idea for the specific problem class. - The "parametric" method is not described in any detail, so I am confused by what it represents.Are these based on a parametric estimate of the distribution, where the parameter samples are aggregated with an average?My overall takeaway from the toy examples (which exhibit skewness and/or multiple modes) is that, with averaging techniques such as mean, the resulting aggregated posterior is a poor representation of the true posterior.However, I have questions about whether such comparisons are fair, since (at least in the case of a known bimodal distribution) averaging techniques are clearly a poor choice.Therefore it feels that the comparison is a bit unfair. - It might be nice to see some discussion about whether taking higher-order moments would suffice with unimodal examples. One of the columns in table 1, logistic regression, should show the likelihood of held-out observations under the posterior (e.g. averaged over posterior examples or based on the posterior mean), as optimal performance on holdout data is the most common performance metric in fitting any logistic regression.While I don't expect the proposed method to perform worse than "average", if it doesn't then this method's complexity may not be justified for such applications. - I'd like to see more discussion of the final examples the "Real datasets" section, as the only real discussion of it is contained in the tiny plots.My takeaway from these is that the proposed method is able to find the best approximation to the posterior (as measured by KL divergence) while performing well - as well as the "average" method (while the average method doesn't approximate the posterior well) - on a test set. - The authors should discuss how they selected delta_p and delta_a; was this done by fitting the model over a grid of values?Is there a general rule of thumb for this? How poorly does their algorithm perform for poorly selected values?Tuning these would add a significant barrier in adoption of this method.The description of the partitioning algorithms could be cleaned up.Further, I had no idea what sentences like "Our PART algorithm consists of space partitioning followed by density aggregation" meant on first reading them, because the terms "space partitioning" and "density aggregation" didn't yet have meeting for me (I realize that there are subsections devoted to this later in section 2).A simple cartoon diagram would probably help significantly. - Feel free to drop sentences like "such partitions are multi scale and related to wavelets [8]" since they are useless to people who don't already know the relation to wavelets and already known by people who do. - The author should put much more effort into making Algorithms 1 and 2 more clear.While I am usually pedantic about careful notation, I think pseudocode (and, as noted above, illustrations) with less emphasis on precise notation might improve the discussion here.A highly-marked up version (with full notation) could go into the appendix.


Robust Spectral Inference for Joint Stochastic Matrix Factorization
Spectral inference provides fast algorithms and provable optimality for latent topic analysis. But for real data these algorithms require additional ad-hoc heuristics, and even then often produce unusable results. We explain this poor performance by casting the problem of topic inference in the framework of Joint Stochastic Matrix Factorization (JSMF) and showing that previous methods violate the theoretical conditions necessary for a good solution to exist. We then propose a novel rectification method that learns high quality topics and their interactions even on small, noisy data. This method achieves results comparable to probabilistic techniques in several domains while maintaining scalability and provable optimality.

Summary of the paper: This paper improves Anchor Word topic models [1,2] with rectifying operations in order to satisfy a newly introduced double non-negativity constraint, which is proved to be necessary by authors. Experimental results present the rectification indeed improves various aspects of topic modeling performance.Quality: Although most contents of the paper feel solid, experimental part can be further improved. Figure 4, in particular, have a lot of missing data points and too small to read it easily. I also want to encourage authors to provide training time comparisons because one of the largest benefit of using [1,2] was computation even though I feel the rectification will not require extreme computation.Clarity: Mostly clear.Originality: I believe authors are the first to employ rectification in Anchor Word topic models. Even their method appears to be a straight-forward extension, authors provided enough discussions and theories that justify their idea in a broad spectrum.Significance: Although their idea will not begin an entirely new topic modeling methodology, I feel their algorithm can be easily employed various NMF models and can be directly utilized in large scale topic modeling as authors suggested in Section 6.[1] Arora, S., Ge, R., & Moitra, A. (2012, October). Learning topic models--going beyond SVD. In Foundations of Computer Science (FOCS), 2012 IEEE 53rd Annual Symposium on (pp. 1-10). IEEE.[2] Arora, S., Ge, R., Halpern, Y., Mimno, D., Moitra, A., Sontag, D., & Zhu, M. (2012). A practical algorithm for topic modeling with provable guarantees. arXiv preprint arXiv:1212.4777.


Principal Differences Analysis: Interpretable Characterization of Differences between Distributions
We introduce principal differences analysis (PDA) for analyzing differences between high-dimensional distributions. The method operates by finding the projection that maximizes the Wasserstein divergence between the resulting univariate populations. Relying on the Cramer-Wold device, it requires no assumptions about the form of the underlying distributions, nor the nature of their inter-class differences. A sparse variant of the method is introduced to identify features responsible for the differences. We provide algorithms for both the original minimax formulation as well as its semidefinite relaxation. In addition to deriving some convergence results, we illustrate how the approach may be applied to identify differences between cell populations in the somatosensory cortex and hippocampus as manifested by single cell RNA-seq. Our broader framework extends beyond the specific choice of Wasserstein divergence.

The proposed approach is motivated by the "Cramer-Wold device", which ensures the existence of a linear projection that differentiates two distributions. The authors apply the Wasserstein metric directly on samples from both distributions, and show favorable theoretical properties of such an approach under reasonable assumptions (such as bounded domain variables). The analysis is extended to distributions that differ only in a few dimensions, by considering sparse projections. The resulting computational problem turns out to be non-convex. The authors propose a semidefinite program relaxation and further pre-processing to estimate a solution. Empirical evaluation is provided on simulated and real datasets, including an application to bioinformatics.The paper is well written and clearly lays out the main ideas. The paper dresses a problem of interest to the community using a novel technique and sufficient theoretical and empirical validation. Minor issues: - I would encourage the authors to include more description and legends in each figure so they are more self contained and simplify interpretation. - the term "Cramer-Wold device" seems to be fairly non-standard outside of the reference [10]. Would the authors prefer the more standard "Cramer-Wold theorem"?Suggestions for future work: The presented theoretical results are concerned with the sample complexity as compared to the optimal linear projection. It would also be useful to characterize the relationship between the induced divergences and more standard divergences computed directly between the high dimensional distributions, as this would help clarify the convergence in distribution implied by the Cramer Wold Theorem.


A Universal Catalyst for First-Order Optimization
We introduce a generic scheme for accelerating first-order optimization methods in the sense of Nesterov, which builds upon a new analysis of the accelerated proximal point algorithm. Our approach consists of minimizing a convex objective by approximately solving a sequence of well-chosen auxiliary problems, leading to faster convergence. This strategy applies to a large class of algorithms, including gradient descent, block coordinate descent, SAG, SAGA, SDCA, SVRG, Finito/MISO, and their proximal variants. For all of these methods, we provide acceleration and explicit support for non-strongly convex objectives. In addition to theoretical speed-up, we also show that acceleration is useful in practice, especially for ill-conditioned problems where we measure significant improvements.

The authors propose a general scheme of wrapping an existing first order gradient method to solve the minimization of the original objective plus an iteratively sharper proximal term. The authors show that for a generic first order method this 'accelerates' the convergence by changes the depends on \frac{L}{\mu} to \sqrt{\frac{L}{\mu}}.Overall, it is a goodpaper with a couple deficiencies in(a) experiments and(b) given the existing algorithms, when exactly would a catalyst be useful ? (especially in ML). Despite the deficiencies, I still vote for acceptance as it has some ideas which could spur future research, for e.g. the idea of doing multiple passes over data with each pass having a sharper proximal term (similar to [8])CommentsPage 2, line 57 - are the authors sure that 'all' the incremental methods have a O(n) cost ? Including ProxSVRG ?The authors need to make it clear when their contributions are useful. For e.g., - when users cannot use the dual, and - ideally when the objective is not strongly convex (saying that n < L/mu seems, in my opinion, is a little weak and less useful regime)- and when data cannot be held in memory (as the inner algorithm can make a sequential passes over data).line 314, redundant 'do'line 390 - what does 'default' parameters ? default as in optimal w.r.t theory ? or just happen to be parameters in some source code ? The authors of SAG also outlined a procedure to do learning rate scheduling - was this taken into account.Was 'L' just set to the upperbound 'L' in the dataset ? was there any effort to tune this ?How was \eta set ? Was there any effort taken to ensure that enough tuning was performed on all the algorithms ? or was it the case that the authors picked a set of values and lucked out by getting a better performance on the proposed scheme ?One of the benefits of using a method like SAG/SAGA/SVRG/.. is to ensure quick initial convergence which is not guaranteed by FG or Acc-FG. It would be nice to see how the proposed method does w.r.t Acc-FG.SuggestionsTheorem 3.3. is misleading at a first glance, could the authors please ensure the contents of line line 256-257 are located in a more accessible place ? like the theorem statement itself.Table 1 caption " To simplify, we only present the case where n <= L/\mu when \mu > 0." could be boldened to ensure that it is read


Linear Multi-Resource Allocation with Semi-Bandit Feedback
We study an idealised sequential resource allocation problem. In each time step the learner chooses an allocation of several resource types between a number of tasks. Assigning more resources to a task increases the probability that it is completed. The problem is challenging because the alignment of the tasks to the resource types is unknown and the feedback is noisy. Our main contribution is the new setting and an algorithm with nearly-optimal regret analysis. Along the way we draw connections to the problem of minimising regret for stochastic linear bandits with heteroscedastic noise. We also present some new results for stochastic linear bandits on the hypercube that significantly improve on existing work, especially in the sparse case.

In this paper, the authors extend the "resource allocation with semi-bandit feedback", proposed by Lattimore et al. [2014], to the multi-resource case. The paper has provided two regret bounds, one for the worst case (Theorem 2) and the other for the "resource-laden" case (Theorem 7). The authors also provide a new result on the "weighted least squares estimation", which is independently interesting.The paper is well-written and very interesting, the analysis in this paper is also rigorous. The extension to the multi-resource case is non-trivial, and the new result on the "weighted least squares estimation" is very interesting and might be reused by researchers in the field of bandit/RL in the future. Thus, I think this paper meets the acceptance threshold.My main concern about this paper is its practical performance. As the authors have pointed out, the optimization problem in Line 8 of Algorithm 2 is likely to be NP-hard, and some approximation algorithms (e.g. "alternative optimization") must be used in practice. It is not clear to me that when such approximation algorithms are used, whether or not the performance of the proposed algorithm (measured in regret) is still good. Specifically, in Section 6, the authors only report experiment results in two baby examples (both with D=K=2), which are not convincing.Two minor comments:1) In this paper, the authors assume that all the resource types have budget 1. It is not clear to me if this "uniform budget assumption" is essential for the results of this paper. Please explain!2) In Line 39-40, the authors assume that the kth row is associated with the kth task, while in Line 361-362, the authors assume that the kth column is associated with the kth task. Please modify the paper to make them consistent.************************************************************************************************************************* The authors' rebuttal has addressed my minor comments. However, it has not addressed my main concern. So I think 6 is an appropriate score.


Streaming Min-Max Hypergraph Partitioning
In many applications, the data is of rich structure that can be represented by a hypergraph, where the data items are represented by vertices and the associations among items are represented by hyperedges. Equivalently, we are given an input bipartite graph with two types of vertices: items, and associations (which we refer to as topics). We consider the problem of partitioning the set of items into a given number of components such that the maximum number of topics covered by a component is minimized. This is a clustering problem with various applications, e.g. partitioning of a set of information objects such as documents, images, and videos, and load balancing in the context of modern computation platforms. In this paper, we focus on the streaming computation model for this problem, in which items arrive online one at a time and each item must be assigned irrevocably to a component at its arrival time. Motivated by scalability requirements, we focus on the class of streaming computation algorithms with memory limited to be at most linear in the number of components. We show that a greedy assignment strategy is able to recover a hidden co-clustering of items under a natural set of recovery conditions. We also report results of an extensive empirical evaluation, which demonstrate that this greedy strategy yields superior performance when compared with alternative approaches.

This paper proposes a simple greedy strategy for the min-max hypergraph partitioning problem under the streaming model. This paper defines a stochastic model for generating the input hypergraph and show the sufficient conditions for the asymptotical recovery of the hidden clusters for the hypergraphs generated using this model.I think the paper studies an interesting and important problem. The proposed stochastic model for graph generation seems interesting and novel. I find it interesting to see that the min-max graph partitioning problem can be almost recovered under certain conditions of the graph. However, I am mostly concerned about how realistic the stochastic model for generating the graph is: does the hidden co-clustering assumption about how the data is generated realistic in practice? I hope to see more justification of why this model is natural for real-world applications.   I think the authors should also cite a recent but very relevant paper below, which essentially considers the same problem of min-max graph partitioning and proposes an offline greedy algorithm very similar to the one proposed in this work. The offline greedy algorithm is claimed to run with time complexity of O(k|E|) , where k is the number of parts in the resulting partitioning and |E| is the number of edges in the bipartite graph. I am curious to see how the streaming greedy algorithm performs relative to the offline greedy algorithm empirically.    Graph Partitioning via Parallel Submodular Approximation to Accelerate Distributed Machine Learning Mu Li, Dave G. Andersen, Alexander J. Smola, 2015http://arxiv.org/pdf/1505.04636.pdfMinor comments:Line 168-171: It seems that this part of paper is trying to claim that the random partitioning of the graph can yield bad performance, but all it shows is that the random partitioning gives a cost of (1-1/e)m, whereas it is unclear what the global optimal is.Description of Alg 1 (Line 216-225): The notations used in Alg 1 are a bit confusing. At the beginning of Alg 1, S_i's are defined to be the assignment of the items, which makes me think that S_i should denote the set of items in block i. However, it seems that S_i is used to refer to the set of topics covered by the items in block i in the description of the algorithm. I find it quite confusing.


A Pseudo-Euclidean Iteration for Optimal Recovery in Noisy ICA
Independent Component Analysis (ICA) is a popular model for blind signal separation. The ICA model assumes that a number of independent source signals are linearly mixed to form the observed signals. We propose a new algorithm, PEGI (for pseudo-Euclidean Gradient Iteration), for provable model recovery for ICA with Gaussian noise. The main technical innovation of the algorithm is to use a fixed point iteration in a pseudo-Euclidean (indefinite "inner product") space. The use of this indefinite "inner product" resolves technical issues common to several existing algorithms for noisy ICA. This leads to an algorithm which is conceptually simple, efficient and accurate in testing. Our second contribution is combining PEGI with the analysis of objectives for optimal recovery in the noisy ICA model. It has been observed that the direct approach of demixing with the inverse of the mixing matrix is suboptimal for signal recovery in terms of the natural Signal to Interference plus Noise Ratio (SINR) criterion. There have been several partial solutions proposed in the ICA literature. It turns out that any solution to the mixing matrix reconstruction problem can be used to construct an SINR-optimal ICA demixing, despite the fact that SINR itself cannot be computed from data. That allows us to obtain a practical and provably SINR-optimal recovery method for ICA with arbitrary Gaussian noise.

This paper addresses the problem of blind signal extraction (recovery) in the presence of Gaussian noise.To solve the noisy ICA problem the authors employ a fixed point iteration to achieve a maximization of directional kurtosis.The fixed-point iteration is carried out in a pseudo-Euclidean space.Based on the proposed "pseudo-Euclidean gradient iteration (PEGI)" two algorithms are presented: one for estimating a single column of the mixing matrix and a second one using an adapted deflation method to find the full mixing matrix while making sure not to find the same component again.clarity:The paper is clearly written and a pleasure to read. However some key references of related work are missing (see below).originality:The approach is similar to the well-known fastICAmethod and the GI-ICA [21] (Voss et al.,NIPS 2013), however it differs from those methods in an important aspect: It does neither rely on the pre-whitening (orthogonalization) step nor on "quasi-whithening" (quasi-orthogonalization).The main contributions(1)gradient-based fixed point iterations for kurtosis maximisation are done in a pseudo-Euclidean space(2) analysis and insight how to choose this pseudo-Euclidean space optimally for the noisy ICA caseare technically sound and sufficiently novel. Significance:Performing ICA in the noise-corrupted setting is an important problem of potential interest to the NIPS crowd.Though the robustness to Gaussian noise is inherent to all higher-order cumulant-based approaches and is well know, the proposed method to integrate this into a fixed-point algorithm is admittedly elegant and thus the contribution can be considered significant. A further plus is the straightforward extension to the complex case (which is shown in supplemental material) The experimental section is a bit weak, only synthetic data is considered and many other robust ICA methods are not included in comparison.At least experimental results demonstrating improvements over GI-ICA indicate that the cumbersome quasi-orthogonalization as in [21] is no longer needed.On the other hand there exist many methods for the noisy ICA problem. In particular, the algorithm in [R2] "RobustICA" would be the real competitor, because as shown in [R3] the fixed-point algorithm can be seen as a special case of gradient descent with an optimally selected stepsize as proposed in [R2] but such a comparison is not included. Also the relative Newton method of Zibulevsky [R5] for minimization of a quasi-ML cost function would have provided a better baseline.A further issue is that the estimation of 4-th order cumulants / kurtosis in general needs a LOT of data (1e+5--1e+6 samples are often needed, as can be seen from figure 2)Furthermore the practical usefulness for real-world data remains to be demonstrated.further comments:-It would be recommended to compare fastICA with pow3 not tanh nonlinearity in Fig1-Why is the iteration index k omitted in algorithm 2 ? additional references:[R1] Sergio Cruces, A. Cichocki, S. Amari, "The Minimum Entropy and Cumulant Based Contrast Functions for Blind Source Extraction", Lecture Notes in Computer Science LNCS-2085, Springer-Verlag, pp. 786-793, 2001.[R2] Vicente Zarzoso and Pierre Comon, "Robust Independent Component Analysis by Iterative Maximization of the Kurtosis Contrast With Algebraic Optimal Step Size" IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 21, NO. 2, FEBRUARY 2010,10.1109/TNN.2009.2035920 [R3] H. Li and T. Adali. A class of complex ICA algorithms based on the kurtosis cost function.IEEE Transactions on Neural Networks, 19(3):408-420, 2008. [R4] Javidi S, Mandic DP, Took CC, and Cichocki A: "Kurtosis-based blind source extraction of complex non-circular signals with application in EEG artifact removal in real-time.", Front Neurosci, 5, 105 (2011)[R5] M. Zibulevsky,"Relative Newton and Smoothing Multiplier Optimization Methods for Blind Source Separation", chapter in the book: S. Makino, T.W. Lee and H. Sawada eds., Blind Speech Separation, Springer Series: Signals and Communication Technology XV, 2007[R6] P. A. Regalia and E. Kofidis, "Monotonic convergence of fixed-point algorithms for ICA," IEEE Trans. Neural Netw., vol. 14, no. 4, pp. 943-949, Jul. 2003.summaryThe authors present an elegant modification of the fixed-point iteration for robust ICA in a kurtosis-maximization framework.The provided insights would be of interest to the ICA community even though the practical usefulness of the method for real-world data remains to be demonstrated.


Distributed Submodular Cover: Succinctly Summarizing Massive Data
How can one find a subset, ideally as small as possible, that well represents a massive dataset? I.e., its corresponding utility, measured according to a suitable utility function, should be comparable to that of the whole dataset. In this paper, we formalize this challenge as a submodular cover problem. Here, the utility is assumed to exhibit submodularity, a natural diminishing returns condition prevalent in many data summarization applications. The classical greedy algorithm is known to provide solutions with logarithmic approximation guarantees compared to the optimum solution. However, this sequential, centralized approach is impractical for truly large-scale problems. In this work, we develop the first distributed algorithm -DISCOVER -for submodular set cover that is easily implementable using MapReduce-style computations. We theoretically analyze our approach, and present approximation guarantees for the solutions returned by DISCOVER. We also study a natural trade-off between the communication cost and the number of rounds required to obtain such a solution. In our extensive experiments, we demonstrate the effectiveness of our approach on several applications, including active set selection, exemplar based clustering, and vertex cover on tens of millions of data points using Spark.

This paper proposes a distributed algorithm DISCOVER for solving submodular set cover problem. The idea of DISCOVER is to iteratively run the GreeDI as an oracle to incrementally obtain a solution until its valuation is greater than the solution quality requirement. The algorithm can tradeoff between the number of iterations and the communication cost needed for each iteration. This method is also validated on large-scale data applications. I think this paper studies an important problem and the proposed algorithm -- DISCOVER is novel under the distributed setting. The paper is well presented. The algorithm is complemented with theoretical justification and the empirical validation is also extensive. But still, I hope the authors could address the some of my questions below.  Some questions and comments:(1) The section about the tradeoff between communication cost and number of rounds is a bit confusing to me. It seems to me that by choosing smaller \alpha, it requires more rounds, but each round needs smaller amount of communication. I think the goal should be to find an \alpha such that the overall communication cost is minimized, instead of only looking at the cost per round. I think it would be interesting to have more discussion on how to choose \alpha in practice. (2) Instead of setting \alpha as a constant between 0 and 1, it may be more efficient to set alpha adaptively as the algorithm proceeds. One possibility could be to set \alpha close to 1 at the first few rounds, and then gradually decreases \alpha to 0 towards the end. (3) In Alg. 3, suppose we set \alpha = 1. I think the tradeoff between the communication cost and the number of rounds may also be achieved by replacing the doubling strategy (Line 8 of Alg. 3) with \ell = \ell * \beta, where \beta > 1 is a hyperparameter that controls the tradeoff. When \beta is only slightly larger than 1, the algorithm takes more rounds with the communication cost of each round being small. (4) In the experiment section, what \lambda is used for each experiment?Minor comment:The bound in Theorem 4.3 may be further improved: the following paper shows a guarantee of(1-1/e)/ (2\sqrt{k}) with k being the cardinality constraint for GreeDI and also shows that GreeDI is tight at 1/\sqrt(k).The Power of Randomization Distributed Submodular Maximization on Massive Datasets http://arxiv.org/pdf/1502.02606v2.pdf


Online Gradient Boosting
We extend the theory of boosting for regression problems to the online learning setting. Generalizing from the batch setting for boosting, the notion of a weak learning algorithm is modeled as an online learning algorithm with linear loss functions that competes with a base class of regression functions, while a strong learning algorithm is an online learning algorithm with smooth convex loss functions that competes with a larger class of regression functions. Our main result is an online gradient boosting algorithm that converts a weak online learning algorithm into a strong one where the larger class of functions is the linear span of the base class. We also give a simpler boosting algorithm that converts a weak online learning algorithm into a strong one where the larger class of functions is the convex hull of the base class, and prove its optimality.

The paper extends the theory of boosting to the online learning setting, proposing strong learning algorithms, building up from weak learners. First of all, I want to say that the topic of online boosting is quite fascinating. Theory of boosting has been of great significance to both statistics and machine learning community over the past two decades. Recently, there has been a couple of papers which has tried to extend boosting for binary classification to the online setting. In fact, the latest paper (by Satyen Kale et al.) won the best paper award at ICML 2015! Significance- As discussed above, the significance of online boosting is beyond question.Originality- As far as I know, no one has published work on extending boosting for regression to the online setting. Definitely original.Quality- Assuming the proofs are correct (I have not checked the analysis of Algo.2 and I have a major concern about the regret bound of Alg.1, which I will come to later), the paper is definitely of good quality.Clarity- In my opinion, the main drawback of the paper is the presentation. I will list as many concerns as I can recollect. 1. Discussion about weak and strong learners- The paper directly gives a definition of weak learning algorithm and strong learning algorithm. What are the intuition behind them? How do they relate to weak and strong learning for regression in the offline case? I am much more familiar with boosting for classification. I know that boosting for regression is approached more from a statistical viewpoint (as greedy forward model selection). Is there a weak and strong learning equivalent in the offline setting for regression?2. Line 117-125- I am slightly perturbed that just 1 paragraph sums up online boosting for regression. It is a very novel concept and in my opinion, requires much more discussion. Why is the strong learner defined for richer regression function class F' and richer loss function class C'? Specifically, why is the loss class changing? What is the equivalent in offline setting? 3. Line 136-138- I did not understand why we cannot have uniform regret bound for all functions in span(F). In fact, if the paper is the online equivalent of Zhang and Yu's paper (as stated by the authors), then the strong learner should converge to the best linear combination of base functions (i.e., min over span(F)).4. The richer loss function class C not only needs to be convex but also Lipschitz and more importantly, smooth. This was not mentioned in the abstract, where it seems the results will hold for all convex functions. 5. Alg.1- Why not have some intuition leading to Alg.1? I had to read the proof to even start understanding what is going on in the algorithm. It is not necessarily a criticism, but I always feel discussions after a very technical algorithm is helpful. As an example, see Kale's paper on online boosting. They lead to their algorithm in a very natural way. 6. Theorem 1- I have a big concern about the theorem. First of all, it seems for the regret to go to 0, N -> \infty. However, for very large N, does the algorithm remain efficient? This is because in step 8, d-dimensional vectors y^i are created for all copies of A (i.e. total of N). Will not soon become too large? More important question- The claim is that average regret goes to 0 as N and T-> \infty. It is not clear what is the rate at which the regret goes to zero. Considering the first term in the regret will go to zero only for \eta= O(log(N)/N) (though it is not clear to me at what rate), the second term is then going to 0 at rate O(log(N)/N), which is slower than even linear (in N) rate. A major advantage of boosting is the rate at which training error goes to zero. Even for online boosting (Kale et al.'s paper), they have error going to zero exponentially fast with N (effectively agreeing with offline boosting, where training error goes to zero exponentially fast with N). So combined with the fact that the regret is not even uniform over the linear span, I would like some clarification of the issue.7. In Alg.1, why cannot we just set B=\etaND? Why need to take a min over some other value? I did not see what it was adding to the algorithm. Line 284 shows the equality and the upper bound in line 297 will come from the Taylor expansion. Though taking a smaller B will give smaller regret, on the other hand, B= \etaND will get rid of the extra \eta^2 \beta B^2 term and make it more readable, isn't it? 8. Why will we care about competing with the convex hull of H? What is the offline equivalent? I can understand competing with best in span(F) but why convex hull?8. In Theorem 2., what does dependence of this regret bound on N is optimal mean? Should I interpret the sentence as : No strong learning algorithm can have a tighter regret in terms of N, against all adversary? There are more places where I felt more discussion could have been added. While I want to emphasize that I strongly believe this is a good piece of work, I think it will be greatly beneficial to the machine learning community if the paper goes through a solid round of editing before acceptance.


Halting in Random Walk Kernels
Random walk kernels measure graph similarity by counting matching walks in two graphs. In their most popular form of geometric random walk kernels, longer walks of length k are downweighted by a factor of λ k (λ < 1) to ensure convergence of the corresponding geometric series. We know from the field of link prediction that this downweighting often leads to a phenomenon referred to as halting: Longer walks are downweighted so much that the similarity score is completely dominated by the comparison of walks of length 1. This is a naïve kernel between edges and vertices. We theoretically show that halting may occur in geometric random walk kernels. We also empirically quantify its impact in simulated datasets and popular graph classification benchmark datasets. Our findings promise to be instrumental in future graph kernel development and applications of random walk kernels.

-weak review-The paper contains a theoretical proof of existence of the phenomenon of halting in geometric random kernels and anextended experimental section that both supports the theoretical results and furthermore has surprising implications on kernel selection in practical contexts. The paper is very well written and never displays any lack of clarity. The math is sound and only draws from rather basic concepts. The paper contains two main contributions: First, the authors provide a mathematical proof that halting can occur in geometric random walk kernels (which had previously been conjectured) by showing that certain kernel parameter choices lead the geometric random walk kernel to degrade into a simple topology-unaware kernel that is a superposition of a vertex label histogram kernel and a vertex-edge label histogram kernel. Secondly, the authors study the effect of halting empirically. For a variety of popular graph classification benchmark datasets, the authors observe halting to occur and are able to quantify its effects. The authors also arrive at a set of practical suggestions wrt reasonable use cases of geometric random walk kernels.Major Comments The paper is very tidy and contains straight-forward reasoning. The main theoretical insight of the authors seems to have been the idea to study the (straight-forward) theoretical relationship between geometric random walk kernels and label histogram kernels. While the theoretical insights are rather trivial, the paper shines particularly in the experimental section: The authors very carefully compare and contrast the effects of kernel selection wrt naive graph kernels, k-step random walk kernels and geometric random walk kernels. In particular, they convincingly conclude that it is in general wise in practice to select a k-step random walk kernel instead of a geometric kernel, even though the latter may naively seem more powerful. It would be nice to shed more light on why Gaussian RBF kernels on label histograms perform so favourably wrt sophisticated kernels.Minor issues Thm 1 proof: "relationship(s) hold(s)" p. 8 last sentence "a (the) fixed-length random walk kernel "The paper contains novel contributions, in particular a simple proof of a conjecture relevant to the ML community. The authors clearly show that their results effect and guide kernel selection in empirical contexts. Therefore, I propose to accept the paper for publication without further hesitation.


Learning with Incremental Iterative Regularization
Within a statistical learning setting, we propose and study an iterative regularization algorithm for least squares defined by an incremental gradient method. In particular, we show that, if all other parameters are fixed a priori, the number of passes over the data (epochs) acts as a regularization parameter, and prove strong universal consistency, i.e. almost sure convergence of the risk, as well as sharp finite sample bounds for the iterates. Our results are a step towards understanding the effect of multiple epochs in stochastic gradient techniques in machine learning and rely on integrating statistical and optimization results.

AbstractThe paper considers the following scenario for leastsquares regression: - the covariate X is Hilbert space H valued - the considered function class is the set of bounded linear functionals over that Hilbert space H - the population least squares risk over H needs to be minimized. To solve this problem a method based on the empiricalrisk is considered that iteratively goes through the samples and performs a gradient step in the corresponding direction. The main theorems establish consistency (Theorem 1) andlearning rates (Theorems 2 and 3) under early stopping.These results are both in terms of excess risk andH-norm (if a minimizer exists). CommentsThe paper nicely describes the regularization effect of early stopping for this particular learning problem. In addition, the considered rather general setupincludes both linear regression and regression with RKHSs (for the latter it would have been nice though, toinclude a computational cost estimate). On the downside, the obtained rates for excess risk are far from beingoptimal. The proofs, which are mostly deferred to an appendix, use Hilbert space valued concentration inequalities, but the way the proof is set up, it seems that significantnew work is needed compared to the case of Thikonovregularization ("kernel least squares regression" or"least squares regression").Here some minor comments:pages 2. It should be better communicated why theHilbert space approach is taken (instead of an RKHS approach). Also, it should be better explained how the latter is recovered from your general setu.page 3, line following (4). I have no idea where an assumption like (3) and (4) can be found in [28]. This needs to be explained in detail.


Asynchronous stochastic convex optimization: the noise is in the noise and SGD don't care
We show that asymptotically, completely asynchronous stochastic gradient procedures achieve optimal (even to constant factors) convergence rates for the solution of convex optimization problems under nearly the same conditions required for asymptotic optimality of standard stochastic gradient procedures. Roughly, the noise inherent to the stochastic approximation scheme dominates any noise from asynchrony. We also give empirical evidence demonstrating the strong performance of asynchronous, parallel stochastic optimization schemes, demonstrating that the robustness inherent to stochastic approximation problems allows substantially faster parallel and asynchronous solution methods. In short, we show that for many stochastic approximation problems, as Freddie Mercury sings in Queen's Bohemian Rhapsody, "Nothing really matters."

Summary of paper:The paper studies a class of asynchronous methods to solving optimization problems in which the objective function is an integral functional of a random process or variable integrated against a probability measure. The class of problem assumed is that the argument of the functional for each non-zero measure set or outcome in the sample space upon which the measure is defined will produce a convex function. In this case the authors demonstrate that asymptotically, completely asynchronous stochastic gradient procedures achieve optimal convergence rates for the solution of convex optimization problems under nearly the same conditions required for asymptotic optimality of standard stochastic gradient procedures. The authors are able to then propose based on these results specialised asynchronous procedures based on multi-core parallelization schemes for stochastic gradient based optimization algorithms which behave analogously to synchronous procedures.Quality:The problem statement and all assumptions are very carefully set out and clearly written and explained with regard to the meaning and significance of such assumptions. The exposition and explanation of Theorem 1 is very well presented as is the technical details for Theorem 2 and the required Lemmas in the appendix.The examples selected to illustrate the method were also very interesting having selected the regression examples based on linear regression as a toy illustration and then non-linear logistic regressions were proposed. These were relatively simple and demonstrated that the required conditions were not overly restrictive for real problems in applications settings.However, it would be perhaps informative to explain a little more background on the context of the extended problems based on stochastic monotone operator problems, this is left very brief and a citation given, it would be useful to see a little more context here.In assumption C and D for the distribution of the noise process it would be good to get more explanation of these assumptions and their significance and interpretation.The experiment performed is very carefully detailed and well designed.Clarity:The paper is highly technical for a NIPS audience but it is very well written and explained carefully so it is still in my opinion in the scope of the conference. I believe the authors have done an excellent job to explain such detailed results in a short paper so clearly and concisely.Originality:The paper is based on several existing works but provides a useful set of results for practical problems which generalise and relax assumptions of existing works.Significance:The authors show that their results significantly generalize existing works that have considered similar results, though as they state previous works of a related nature often impose strong conditions on gradient sparsity or conditioning of the Hessian or allowable types of asynchrony. In particular they only require minimal conditions which are that at the argument that minimizes the objective function one has a positive definite evaluation of the function and that the gradient are smooth.


Supervised Learning for Dynamical System Learning
Recently there has been substantial interest in spectral methods for learning dynamical systems. These methods are popular since they often offer a good tradeoff between computational and statistical efficiency. Unfortunately, they can be difficult to use and extend in practice: e.g., they can make it difficult to incorporate prior information such as sparsity or structure. To address this problem, we present a new view of dynamical system learning: we show how to learn dynamical systems by solving a sequence of ordinary supervised learning problems, thereby allowing users to incorporate prior knowledge via standard techniques such as L 1 regularization. Many existing spectral methods are special cases of this new framework, using linear regression as the supervised learner. We demonstrate the effectiveness of our framework by showing examples where nonlinear regression or lasso let us learn better state representations than plain linear regression does; the correctness of these instances follows directly from our general analysis.

Summary -------The authors present a new algorithm for learning dynamical systems, which generalizes some existing spectral learning algorithms. The approach is based on keeping track of the latent state via a observable representation, which can be updated using a linear mapping for the model classes consider in the manuscript (eg HMMs, linear dynamical systems). To arrive at an unbiased estimate of this mapping from data the authors apply instrumental variable regression.  Comments --------The article contains interesting contributions to the literature on learning dynamical systems. Basic versions of the idea of formulating dynamical system learning as supervised learning problems have be, but the paper gives useful generalizations of the approach and proves error bounds. The main drawbacks are the relative simple numerical experiments and that the manuscript feels a little hastily written. The latter might have contributed to me understanding the approach less well than I would like to. - A main advantage of the proposed approach, that the authors point out, is that the formulation of the learning procedure as regression problems allows to tap into the regression literature for improving and generalizing the algorithm. However, I don't understand why this should mostly apply to the stage 1 regression. E.g. the experiment presented in 5.2: I'm guessing in this case the $W$ is closely related to the transition matrix $T$. Wouldn't it make sense to try to learn a structured $W$ that somehow captures the block-structure of the prior, instead of using a L1 regression in the S1 pre-processing step. Also, in a non-linearity dynamical system (say with linear Gaussian observations), would we want to use a non-linear regression for S2? What am I missing here? This should be clarified in the manuscript.- Sec 5.1: This a very simple model to learn. It would be much more convincing if the authors also presented a slightly more high-dimensional case. Also, how does naive EM-HMM learning perform here; this comparison would be quite instructive.- Sec 5.1: The authors should explicitly spell out why their algorithm gives better performance in this case, and why model 3 performs better than model 2.  - Sec 5.1: Why is logistic regression a sensible choice here? This is an important demonstration for the method, so the manuscript would benefit from a more detailed explanation/justification. If the authors need more space, I suggestion possible deleting figure 1.- The authors should cite some of the traditional system identification literature, where instrumental variables have been used for a long time, eg [Ljung. System identification, 1999] and references therein and explain where the similarities and differences lie.- Is the regression stage S1B really necessary? If so, why? - minor points: -- l25: missing "of"-- l79: missing ")" -- l94: It would be useful if you could briefly remind the reader of the definition of k-observability and how this affects the choice of future feature $\psi_t$ -- l99: maybe replace "is determined" by "can be computed from" -- l121: Should it be conditioned on $o_{t-k:t+k-1}$ -- l190: typo, should prob be $P(o_{t:t+k-1}\vert o_{1:t-1})$ -- table one, top left box: there's some confusion with the indices here. For a stationary HMM, $P(o_2,o_1)$ and$P(o_t,o_t)$ are the same, but a consistent notation should be used.


COEVOLVE: A Joint Point Process Model for Information Diffusion and Network Co-evolution
Information diffusion in online social networks is affected by the underlying network topology, but it also has the power to change it. Online users are constantly creating new links when exposed to new information sources, and in turn these links are alternating the way information spreads. However, these two highly intertwined stochastic processes, information diffusion and network evolution, have been predominantly studied separately, ignoring their co-evolutionary dynamics. We propose a temporal point process model, COEVOLVE, for such joint dynamics, allowing the intensity of one process to be modulated by that of the other. This model allows us to efficiently simulate interleaved diffusion and network events, and generate traces obeying common diffusion and network patterns observed in real-world networks. Furthermore, we also develop a convex optimization framework to learn the parameters of the model from historical diffusion and network evolution traces. We experimented with both synthetic data and data gathered from Twitter, and show that our model provides a good fit to the data as well as more accurate predictions than alternatives.

This paper extends previous work on modeling network interactions with multivariate Hawkes processes by introducing a time-varying network into the model. For example, when one Twitter user publishes a tweet, followers of that user are likely to retweet in response. The dynamic network is intended to capture the creation of new connections, for example, when one Twitter user begins to follow another. The instantaneous network is represented by a binary adjacency matrix, and edges are added to the network according to a survival process with a event-driven rate. If one user frequently retweets another user's messages, then it is likely they will begin to follow that user and thereby add a new connection to the network. This combined modeling of activity and network evolution is captured in a joint probabilistic model, and inference is performed by maximum likelihood estimation.The problem is well-posed and elegantly solved. The paper could be strengthened by addressing a few concerns. First, it appears that the interaction weights ($\beta_s$'s) are shared by all interactions originating from source $s$. Intuitively, it seems that weights should instead be connection specific since the influence of a source node may vary from destination to destination. That is, it seems that $A$ should be accompanied by a weight matrix $W$. Second, this problem is conceptually related to the problem of inferring time-varying connection strengths given observations of spiking activity, which has received particular attention in the context of modeling neural synaptic plasticity. See Stevenson and Kording (NIPS 2011), Linderman, Stock, and Adams (NIPS 2014), and Song et. al. (J. Neuro. Meth. 2015). These papers model the evolution of a real-valued weight matrix rather than a binary adjacency matrix. Third, though in the context of Twitter it may be reasonable to assume that events are marked with their source and destination, in more general scenarios we may not observe the source. Your model could handle this scenario with the addition of latent variables for the unknown source, as in Simma and Jordan (UAI 2011). Notes: - The legend in Figure 6 says "Hawks" instead of "Hawkes"- "Twitter" and "Hawkes" should be capitalized in references - Linderman and Adams reference should be to ICML 2014 rather than arxiv* UPDATE * I appreciate the authors responses. I do think some of those clarifications/generalizations could be worthwhile additions to the paper.


Solving Random Quadratic Systems of Equations Is Nearly as Easy as Solving Linear Systems
This paper is concerned with finding a solution x to a quadratic system of equations y i = | a i , x | 2 , i = 1, . . . , m. We demonstrate that it is possible to solve unstructured random quadratic systems in n variables exactly from O(n) equations in linear time, that is, in time proportional to reading the data {a i } and {y i }. This is accomplished by a novel procedure, which starting from an initial guess given by a spectral initialization procedure, attempts to minimize a nonconvex objective. The proposed algorithm distinguishes from prior approaches by regularizing the initialization and descent procedures in an adaptive fashion, which discard terms bearing too much influence on the initial estimate or search directions. These careful selection rules-which effectively serve as a variance reduction scheme-provide a tighter initial guess, more robust descent directions, and thus enhanced practical performance. Further, this procedure also achieves a nearoptimal statistical accuracy in the presence of noise. Empirically, we demonstrate that the computational cost of our algorithm is about four times that of solving a least-squares problem of the same size.

Summary: The paper is concerned with solving a quadratic system of equations. Such problem, appears in a variety of applications, ranging from physics (phase retrieval), to mixed regression (which is of certain interest to the NIPS audience). In fact, the authors propose a quite general way of minimizing a non-convex objective function relying on truncated versions of its gradient (truncated Wirtinger flow), and this may have other implications as in matrix completion or dictionary learning.            Quality: The paper is a very nice piece of work.            Clarity: The paper is well-written, though the technicalities could not be included for obvious space limitations, but can be found in the supplemntary material (and the arxiv version).                  Originality: Phase retrieval via Wirtinger flow (WT) was already propose in [14]. In this paper, the authors propose key algorithmic changes based on a truncated version of WF (TWF) yielding enhanced performance compared WF. TWF operates only upon a subset of data enabling better descent directions, and moves in a more aggressive way in these directions. Under Gaussian design, the approach is shown to provably optimal (in sample complexity, reaching the information theoretic limit) in the noiseless case, stable to noise, and runs in linear-time complexity.             Comments:     1) Figure 5 (a) and (b): in the text, the corresponding experiments are in the noiseless setting. So speaking of Poisson and Gaussian likelihood in the legend is meaningless. Please correct.     2) Figure 5: (b) should be complex Gaussian design.     3) Figure 5 and page 8, line 410-413: the slope should rather be -1/2 because of the square-root on SNR.     4) I found that a clear discussion of the theoretical value of the universal constant c_0 is missing, and should be included and compared to the empirical bounds 5 (real) and 4.5 (complex) claimed in the text.


Rectified Factor Networks
We propose rectified factor networks (RFNs) to efficiently construct very sparse, non-linear, high-dimensional representations of the input. RFN models identify rare and small events in the input, have a low interference between code units, have a small reconstruction error, and explain the data covariance structure. RFN learning is a generalized alternating minimization algorithm derived from the posterior regularization method which enforces non-negative and normalized posterior means. We proof convergence and correctness of the RFN learning algorithm. On benchmarks, RFNs are compared to other unsupervised methods like autoencoders, RBMs, factor analysis, ICA, and PCA. In contrast to previous sparse coding methods, RFNs yield sparser codes, capture the data's covariance structure more precisely, and have a significantly smaller reconstruction error. We test RFNs as pretraining technique for deep networks on different vision datasets, where RFNs were superior to RBMs and autoencoders. On gene expression data from two pharmaceutical drug discovery studies, RFNs detected small and rare gene modules that revealed highly relevant new biological insights which were so far missed by other unsupervised methods. RFN package for GPU/CPU is available at http://www.bioinf.jku.at/software/rfn.

The authors present an intriguing take on factor analysis: namely, the use of Ganchev et al's posterior regularization to enforce non negativity constraints on the posterior. They present proofs of convergence and correctness, and present a scalable method for inference and learning in stacked constrained factor analysis models.The method appears to be sound and is an interesting direction overall, a refreshing departure from much of the existing literature, and the first instance of which I am aware that posterior regularization has been highlighted in a deep learning/unsupervised feature learning context. While I did not review it in detail, the degree of thoroughness demonstrated by the supplementary material is truly impressive.My main concern with this paper is one of being somewhat underwhelmed by the empirical evaluation, in light of the norms of the community. It makes the contribution difficult to judge from that vantage point.In particular, one question I'm left with is whether there is a reason that these models, at least in the one layer case (and probably also in the multi-layer case) cannot be evaluated in terms of their test set likelihood. The quantitative evaluation of unsupervised methods on MNIST, for example, is usually phrased in terms of test set likelihood. Synthetic data results are presented for a set of chosen metrics meant to demonstrate the effectiveness of the model at satisfying the design goals of RFNs, but are difficult to interpret. The classification baselines employed also seem to be a bit dated. For example, Komer et al (2014) report a test set error of 11.7% on CONVEX using a combination of a polynomial SVM and PCA preprocessing. While that result is relatively hard to find, unsupervised learning on CIFAR, for example, has progressed quite a bit (albeit in some cases with domain-knowledge-heavy architectures).- "Current unsupervised deep learning approaches like autoencoders or restricted Boltzmann machines (RBMs) do not model specific structures in the data." This is unclear. It seems that the filters of individual units can model "specific structures". The next sentence refers to "generative models" but RBMs and (deep belief networks built from them) _are_ generative models. More precision is required in this section about what is meant. - I'm curious as to the authors' response to the success of representations that eschew sparsity altogether and still manage to perform very well, such as maxout networks (Goodfellow et al, 2013). - More elaboration on the role of normalizing constraints would be helpful in exposition. - I assume the "projected Newton method" is the Newton step projected onto the constraint surface in terms of the closest (in terms of L2 distance) point obeying the constraint. It would be good to explicitly state this.POST-REBUTTAL COMMENTS: - The argument against using likelihood is somewhat compelling though it seems like in a deep model one should be able to recover the lost ground caused by posterior regularization in a shallow model. - I'd suggest moving the CIFAR results front and center along with the drug design stuff, and perhaps relegating the table presented to the supplementary. CIFAR10/100 being a point of contact for many in the deep learning world I think highlighting that in the main text is helpful.I've upgraded my score to a 7.


Regularization-Free Estimation in Trace Regression with Symmetric Positive Semidefinite Matrices
Trace regression models have received considerable attention in the context of matrix completion, quantum state tomography, and compressed sensing. Estimation of the underlying matrix from regularization-based approaches promoting low-rankedness, notably nuclear norm regularization, have enjoyed great popularity. In this paper, we argue that such regularization may no longer be necessary if the underlying matrix is symmetric positive semidefinite (spd) and the design satisfies certain conditions. In this situation, simple least squares estimation subject to an spd constraint may perform as well as regularization-based approaches with a proper choice of regularization parameter, which entails knowledge of the noise level and/or tuning. By contrast, constrained least squares estimation comes without any tuning parameter and may hence be preferred due to its simplicity.

The paper studies the estimation of low-rank semidefinite matrices from noisy inner products with measurement matrices X_1, ..., X_n. Nuclear norm regularized least squares is the dominant approach to estimation in this setting. However, if \Sigma is forced to be positive semidefinite, the nuclear norm reduces to the trace (a single linear function). The paper proves that under some circumstances, one can accurately estimate \Sigma by simply solving a constrained least squares problem (dropping this linear regularization). This is not possible for general Gaussian measurements, but for measurements that do not have semidefinite matrices in their nullspace (e.g., if the X_i are themselves full rank semidefinite matrices), it is, under conditions. Simulations verify the claims. The paper also presents a real data experiment, which appears to be somewhat artificial, but again agrees with the theory presented here. The goal of the paper is to argue that regularization may not be necessary for certain sampling models. How do the theoretical results compare to the best known theoretical results for the regularized problem? The restricted eigenvalue condition is necessary for both formulations; it seems that the separability and correlation conditions are only necessary for the constrained least squares. Comments in the response would be helpful. Suppose we specialize the results to some random measurement ensemble (e.g., X_i = z_i z_i^*, with z_i a random vector, or X_i a Wishart matrix). Is it possible to evaluate \mu, \tau and \phi, and compare the bound in Theorem 2 to the corresponding result for the nuclear norm regularized variant (11)? Comments in the response would be helpful. In a similar vein, while it is easy to recognize that the results in this paper are more general than those of [4]/[10], it would be great if we could recover the rates of the special case as a consequence of the general theory introduced here. A comparison would help to understand the strength of the results. Comments in the response would be appreciated. The main practical implication of the work seems to be that in some situations, it is not necessary to choose a regularization parameter. From a computational perspective, there don't seem to be obvious reasons to prefer (7) to (11). The paper contains solid results, which seem somewhat under-interpreted. I'm eager to hear the authors responses on the above points, as well as some questions below about the practical motivations for the sampling model.  EDIT: after considering the authors responses on the comparison to nuclear norm minimization and the motivations for the sampling model, I have raised my score to a 6.  Smaller issues:Can \mu(T) be controlled in terms of \phi(T) and \tau(T)? A similar correlation quantity shows up in the intermediate steps of classical compressed sensing proofs, where it is bounded in terms of restricted isometry constants. The paper contains a real data experiment showing the estimation of the best rank-r approximation of data covariance, based on inner products with random Wishart matrices. The observations seem to conform (more or less) to the theoretical claims of the paper. However, I'm a little bit puzzled as to the motivation for this experiment. There does not seem to be much practical motivation for this kind of sampling with this kind of data. The message of the paper would be much stronger if it could demonstrate the performance of the proposed method in the context of an application where we really are stuck with a few trace samples. The results require special measurements which do not have semidefinite matrices in their nullspace (in a quantitative sense). "Generic" gaussian measurements do not satisfy this property, but Wishart measurements do. Besides phase retrieval (where this is a somewhat natural model) are there other practical scenarios where trace sampling with semidefinite X_i applies?


Tensorizing Neural Networks
Deep neural networks currently demonstrate state-of-the-art performance in several domains. At the same time, models of this class are very demanding in terms of computational resources. In particular, a large amount of memory is required by commonly used fully-connected layers, making it hard to use the models on low-end devices and stopping the further increase of the model size. In this paper we convert the dense weight matrices of the fully-connected layers to the Tensor Train [17] format such that the number of parameters is reduced by a huge factor and at the same time the expressive power of the layer is preserved. In particular, for the Very Deep VGG networks [21] we report the compression factor of the dense weight matrix of a fully-connected layer up to 200000 times leading to the compression factor of the whole network up to 7 times.

Fully connected dense layers (W x + b) can be a nasty issue for neural networks since they require storing and working with MxN parameters, and this is often a bottleneck for producing larger models. However past work has shown that there is redundancy in this representation. This motivates decomposing the matrix W into smaller components, but there are several difficulties with doing this in practice.In this paper, the authors utilize a general tensor decomposition form known as "Tensor Train" (Oseledets, 2011). They then describe and implement a standard dense layer utilizing this decomposition. The decomposition has several nice properties: 1) there are fast algorithms for running standard linear algebraic ops, and these can be used to implement f/b-prop, 2) the method is numerically stable even decomposed into many parts. They then use this method to decompose a dense layer into a much smaller number of parameters.The main result of the work is shows that a dense layer in vgg can be replaced with a decomposed layer with a very large reduction in parameters and only a small reduction in accuracy, with no reduction in speed.This is a very large improvement of a baseline method that just learns a low-rank representation. This seems like a impressive and useful result. The paper itself could be much improved in terms of clarity. Some issues:- Generally the figure and table captions are very hard to read and leave several important- Figure 1 in particular is very hard to read and decipher.- Having everything in the paper start with TT- becomes confusing. Naming could be simplified- There are several typos and grammatical issues, be sure to read over closely.  Questions:- How does Tensor train compare here to other tensor decomposition formats?


Variational Dropout and the Local Reparameterization Trick
We investigate a local reparameterizaton technique for greatly reducing the variance of stochastic gradients for variational Bayesian inference (SGVB) of a posterior over model parameters, while retaining parallelizability. This local reparameterization translates uncertainty about global parameters into local noise that is independent across datapoints in the minibatch. Such parameterizations can be trivially parallelized and have variance that is inversely proportional to the minibatch size, generally leading to much faster convergence. Additionally, we explore a connection with dropout: Gaussian dropout objectives correspond to SGVB with local reparameterization, a scale-invariant prior and proportionally fixed posterior variance. Our method allows inference of more flexibly parameterized posteriors; specifically, we propose variational dropout, a generalization of Gaussian dropout where the dropout rates are learned, often leading to better models. The method is demonstrated through several experiments.

** Updated review after authors' response **I spent some time trying to get a broader view of the work since I wrote my review. To be honest, my view now is that the paper is very well written, but with almost no novel contribution. The main idea is "sample a new random variable realisation for each data point in the mini-batch". I have been using this technique for most of my applications working with SVI - it is fairly obvious (although the theoretical variance analysis is novel as far as I know). It does have some computational limitations that the authors avoid by the re-parametrisation trick - which I'm afraid is *very* problem specific. I can't easily see how this trick could be used for models other than MLPs. The author's answer "We show that dropout can be interpreted as being as a result of, rather than being equal to, a local reparameterization of weight uncertainty" suggests that the authors claim dropout to be a special case of the technique; this would have been fine if other, non-trivial, models were used as well. This is my main concern, that the paper would have been *a lot stronger* (thus beneficial to the community) if the authors were to suggest and assess this with general models (rather than models that make use of it anyway, see next). The authors do mention in the rebuttal that this should be applicable to "a larger family of graphical models with plates" which I think is what we would have been most interested in. My second major concern is that the models used in the assessment use this trick anyway. Even though the authors claim in the rebuttal that earlier methods do not use the technique, I am certain that all implementations of dropout (with all its variants) sample new random variable realisations for each point in the mini-batch. I spent a lot of time studying different open-source implementations and, even though this is sometimes overlooked in the literature, the actual implementation would do this. The dropout journal paper mentions this explicitly, and the popular Caffe open source framework implements this as well. Another example is the increasingly popular Keras framework which implements Gaussian multiplicative noise as well as other noise layers, all of which sample new realisations for each data point (https://github.com/fchollet/keras/blob/master/keras/layers/noise.py). I would note that fast dropout operates differently and matches the first two moments of the input. That's why I find the experiments not satisfying. The dropout MLP models are presumably just the vanilla models used in the wild. The main improvement is the optimisation of the dropout parameter. But this was already done before in [A Bayesian Encourages Dropout, Maeda] and can't be counted as a novel contribution (for that matter Shin-ichi also discusses a Bayesian interpretation of dropout). The novel contributions of the paper seem to be1. "sample separate random variable realisations per data-point in stochastic variational inference to reduce the estimator variance" (which I think is important, but somewhat known) and2. an explanation for dropout's low variance optimisation objective (although it might be just my interpretation as I believe this is not mentioned explicitly as an insight for dropout from the analysis).After a long deliberation I have decided to reduce my score slightly as the paper does not communicate these contributions well in its current form. Even though the paper is written well (in the sense that it is very pleasant to read), both the authors and the community would have benefited greatly from re-structuring the paper to present these more focused ideas. ** Review before authors' response **  The main results of the work are very interesting. The analysis (both theoretical and experimental) of the estimator variance of both fast dropout and multiplicative Gaussian noise dropout is important and enlightening.  Major criticism: * The main contribution of the paper is linking multiplicative Gaussian noise dropout and fast dropout to variance reduction techniques in a Bayesian setting, as is mentioned explicitly in the introduction ("in this paper we show that a relationship between dropout and Bayesian inference can be extended and exploited [...]").This is stretched rather thinly as a new technique for gradient based variational inference ("we explore an as yet unexploited trick for drastically improving the efficiency of stochastic [...]") even though the "trick" mentioned is known and has been used for quite some time: both multiplicative Gaussian noise dropout and fast dropout perform this, and this is not new by itself. I would have expected to see the authors use this technique for new models in the experiments section rather than presenting this as a new development and then assessing this with models that *already* use this technique.* I'm confused about the generality of the method suggested. In line 107 it is mentioned that the method can be applied to any probabilistic model, which is true for sampling a new random variable for each data point (as is often done anyway). But the computationally efficient form in line 171-180 is only applicable to neural networks. Furthermore, experiments are only done for neural networks, with all models compared simply being multiplicative Gaussian noise dropout and fast dropout with the only novelty being the tuning of the dropout parameter. * section 3.3: the use of a log-uniform prior seems quite forced (mostly to get rid of the KL divergence term as a function of theta). Especially given that dropout is often used with L2 regularisation in many models, which would add additional unaccounted-for theta terms to the objective function. Furthermore, the fact that the authors have to limit the value of alpha (section 3.4) shows that the effects of the regularisation are not sufficient. Minor criticism: * The citation format is incorrect. * Lines 185-186: It is not clear whether by a "local re-parameterisation trick" the authors mean sampling a separate random variable for each data point in the mini-batch, or specifically sampling a separate random variable for each *unit* in a network. * Lines 57-59: this is wrong; [1] has shown identical results to dropout in a Bayesian setting. * Line 144: why is cov positive on average?* Line 404: why wasn't the dropout probability p scaled down for the smaller networks? what are the alpha values used with fast dropout and multiplicative Gaussian noise dropout? * Lines 410-413: why wasn't Variational (A2) assessed for CIFAR-10 as well? I hope the authors will find the following minor comments helpful: * Many citations have proper names in lower case (such as fisher, bayesian, langevin, etc). You can fix this by surrounding these words with curly brackets in the bib file ("... fisher ..." -> "... {F}isher ..."). * Line 61: "data(Kingma" * Line 100: g() should be f() * Line 112: "are correct on average" is quite informal * Line 132: why does the equation contain the term L_D(phi)^2/N^2 instead of E(L_i)^2? * Line 215: "with a M x K of..." should be "with an M x K matrix of..." * Line 216: "by adding noise ... the weights are less likely to overfit" - why?* Line 233: Wang and Manning (2013)'s derivation uses Bernoullis on the inputs to the product, and the marginal was *approximately* Gaussian. Presumably the reason for the good performance for both dropout and fast dropout is that both Gaussians and Bernoullis had the low variance discussed in this paper (compared to distributions over the weights); fast dropout simply approximated the output of the Bernoulli times weights. * Line 239: "activation noise" is not defined * Line 255-257: a bit confusing * Line 262: "... for which the noise enters this way" is not clear * Lines 290-292: can be moved to appendix * Line 303: "performance is quite robust to different specifications of alpha" - presumably because the model will simply adapt the weights to match the moments of the full posterior? * Line 318: \ref{} should be \eqref{} for equation 3 * Line 330: "similar to our local reparameterization trick" - not so much; Bayer simply matches the first two moments of the input distribution in a variational setting, equivalent to fast dropout with generalised noise processes. * Section 4: you might consider adding references to [2], [3] * Line 362: "lower" should be "lowest" * Lines 371-373: it is not clear that the three models correspond to random variable samples per unit / samples per weight / samples per batch (or data point?) References:[1] Weight Uncertainty in Neural Network Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, Daan Wierstra, ICML, 2015 http://jmlr.org/proceedings/papers/v37/blundell15[2] A Bayesian Encourages Dropout Shin-ichi Maeda Under review as a conference paper at ICLR 2015 http://arxiv.org/pdf/1412.7003.pdf[3] Dropout As A Bayesian Approximation: Insights And Applications Yarin Gal, Zoubin Ghahramani Deep Learning Workshop, ICML, 2015https://sites.google.com/site/deeplearning2015/33.pdf?attredirects=0


Optimal Testing for Properties of Distributions
Given samples from an unknown discrete distribution p, is it possible to distinguish whether p belongs to some class of distributions C versus p being far from every distribution in C? This fundamental question has received tremendous attention in statistics, focusing primarily on asymptotic analysis, as well as in information theory and theoretical computer science, where the emphasis has been on small sample size and computational complexity. Nevertheless, even for basic properties of discrete distributions such as monotonicity, independence, logconcavity, unimodality, and monotone-hazard rate, the optimal sample complexity is unknown. We provide a general approach via which we obtain sample-optimal and computationally efficient testers for all these distribution families. At the core of our approach is an algorithm which solves the following problem: Given samples from an unknown distribution p, and a known distribution q, are p and q close in 2 -distance, or far in total variation distance? The optimality of our testers is established by providing matching lower bounds, up to constant factors. Finally, a necessary building block for our testers and an important byproduct of our work are the first known computationally efficient proper learners for discrete log-concave, monotone hazard rate distributions.

This manuscript aims to the determine the number of observations required to distinguish an unknown distribution $p$ from some class of distributions $\mathcal{C}$ that is at least $\epsilon$ away in total variation distance. Different classes of $\mathcal{C}$ considered include monotone, log-concave, unimodal and moonotone hazard rate distributions.Below are a few questions and comments which will hopefully help the author(s) improve the manuscript:A. Motivations I feel that the motivations for imposing shape-related constraints on discrete distributions remain to be explained further in this manuscript. Discrete distributions are mainly useful for modelling (i) the frequencies of observed objects (e.g. IP addresses) and (ii) counting data. In (i), applying monotone constraint means that one has to impose an ordinal structure on the objects; furthermore, one needs a metric associated with the ordinal structure to use the log-concave constraint. These implications somewhat limit the use of the proposed method, since not all the objects have a natural order.B. Theory. B1. Proof of Theorem 1.In order to have (1) and (2), one needs independence between the observations $X_i$ and the newly-drawn distribution $q$ (which satisfies Properties 1 and 2). This is not always the case, unless the observations used to estimate $q$ are discarded for the rest of the analysis. This requirement should be stated more explicitly.  B2. Use of Lemma 5.In the monotone case, Lemma 5 states that one can find $q$ such that $\mathbb{E}[\chi^2(p,q)]\le \epsilon^2/500$. However, in order to invoke Theorem~1, one requires $\chi^2(p,q) \le \epsilon^2/500$. So there is a gap in between. B3. Continuous case vs discrete case.Birges (1987) was cited a few times in the development of the theory. To my knowledge, the work of Birges only deals with density estimation. Could the author(s) state the particular result from Birges (1987) that has been referred to many times in the manuscript?B4. Log-concavity vs unimodality.In the proof of Lemma 7, as well as in Section H.2 in the appendix, it is stated that `any log-concave distribution is unimodal'. However, this is false in view of the definition given in Section~2 (i.e. $f_{i-1}f_{i+1} \le f_i^2$). For example, consider $n=7$, $f_1 = f_4 = f_7 = 1/3$ and $f_2 = f_3 = f_5 = f_6 = 0$. B5. Rate for testing log-concavity.It is stated in the abstract that testing log-concavity requires $O(sqrt{n}/\epsilon^2)$ samples. However, in view of Theorem 4, this is true when $\epsilon > n^{-1/4}$. Clearly, this statement is not precise when we are in the region of `fixing $n$ and decreasing $\epsilon$'. C. Connections to Statistics. It is worth pointing out that here the author(s) assumed that the distribution of interest, i.e. $p$, is discrete. In addition, in the development of theory, the support of $p$ is assumed to be $[n]$ (i.e., $\{1,2,\ldots,n\}$), which grows with respect to $n$. This setting is quite popular in Computer Science, though perhaps is not very well-known in Statistics. The author(s) cited work such as Hall and Van Keilegom (2005) and Cule and Samworth (2010). However, the cited work only deal with the continuous case. For more relevant work on estimating/testing a discrete monotone and log-concave distribution, see Jankowski and Wellner(2009) and Balabdaoui et.al.(2013). ReferencesJankowski, H. and Wellner, J.A. (2009). Estimation of a discrete monotone distribution. Electronic Journal of Statistics, 3, 1567--1605.Balabdaoui, F., Jankowski, H., Rufibach, K. and Pavlides, M. (2013). Asymptotic distribution of the discrete log-concave MLE and related applications, Journal of Royal Statistical Society Series B, 75, 769-790.


Learning Wake-Sleep Recurrent Attention Models
Despite their success, convolutional neural networks are computationally expensive because they must examine all image locations. Stochastic attention-based models have been shown to improve computational efficiency at test time, but they remain difficult to train because of intractable posterior inference and high variance in the stochastic gradient estimates. Borrowing techniques from the literature on training deep generative models, we present the Wake-Sleep Recurrent Attention Model, a method for training stochastic attention networks which improves posterior inference and which reduces the variability in the stochastic gradients. We show that our method can greatly speed up the training time for stochastic attention networks in the domains of image classification and caption generation.

This paper presents a new learning algorithm for stochastic attention models, employing some recent advances from the literature on training generative models: specifically (1) a simultaneously trained inference network; (2) importance sampling to estimate gradients (a la "reweighted wake sleep"); and (3) tailored control-variates to reduce estimator variance.The use of control variates and (to a lesser degree) a trained inference network are shown to give significant speed-ups to training time.Using this learning algorithm, the model is shown to be capable of achieving comparable performance to a recent variational approach [Mnih et al. ICLR2015] (in it's naive form), but with shorter training times.The authors commendably pack a lot of detail into a small number of pages, and the overall body of work is impressive.However, I was left with some minor questions (see below) about details of the results -- which hopefully the authors can address in their feedback. == Overall the paper has good quality, and it uses a nice combination of sophisticated techniques to achieve good results on interesting problem domains.Equation (10) was also informative/interesting; in the limit, the proposed method gives at least as accurate a bound on the objective as previous variational approaches.One area of confusion that I have is in regards to the performance in comparison to the variational method of Mnih et al, with respect to modifications on top of that model that encourage exploration. It seems that when compared that the variational approach *without* exploration the proposed method shows a significant speed-up. However, with Figure 3 it appears the the method of Mnih et al *with* exploration significantly out-performs (in terms of speed) the proposed approach (and the proposed approach does not seem to benefit from exploration). Is this the case? And if so, why might this be the case, and are there analogous exploration-boosting methods that one could/should consider for the case of the Flickr 8K example in Figure 4? Granted space is limited -- but it would be useful if the authors could address these issues more thoroughly.Also, for the learning curves presented, it might also be informative to show testing/validation error progress in addition (or instead of) training progress.-- In terms of clarity, the paper is reasonably clear and well written. However, there are several parts that could be improved in this regard (in addition to the questions above).One general comment here is attention to detail, and precision with notation. There are several parts of section for in particular where notation usage is a little sloppy. With a little thought from the reader, it's possible to figure out the precise meanings but it would be useful if the authors could make everything explicit. (As an example of this, the expectation distribution in (9) and (10) is not specified, however it is in (13), (15), etc. Another example is the use of set notation {a_n} and an unadorned "a" symbol both represent a sequence of actions.Similarly, it's possible to infer the legends of the figures but this could be made clearer and more explicit in several cases.-- This work has moderate originality. It is a sensible and non-trivial combination of models and techniques that have been proposed recently in the literature. -- Likewise, I think the work has moderate significance. It represents (potentially) algorithmic improvements/speed-ups to an interesting model, using techniques that are currently gaining favour in the community. It remains to be seen whether the methods proposed here have real-world impact (improvements) any significant applications, and whether they help consistently move forward the state of the art. It does, however, represent a useful exploration of the overall model/algorithm design space and in this sense I found it quite valuable.


BinaryConnect: Training Deep Neural Networks with binary weights during propagations
Deep Neural Networks (DNN) have achieved state-of-the-art results in a wide range of tasks, with the best results obtained with large training sets and large models. In the past, GPUs enabled these breakthroughs because of their greater computational speed. In the future, faster computation at both training and test time is likely to be crucial for further progress and for consumer applications on low-power devices. As a result, there is much interest in research and development of dedicated hardware for Deep Learning (DL). Binary weights, i.e., weights which are constrained to only two possible values (e.g. -1 or 1), would bring great benefits to specialized DL hardware by replacing many multiply-accumulate operations by simple accumulations, as multipliers are the most space and powerhungry components of the digital implementation of neural networks. We introduce BinaryConnect, a method which consists in training a DNN with binary weights during the forward and backward propagations, while retaining precision of the stored weights in which gradients are accumulated. Like other dropout schemes, we show that BinaryConnect acts as regularizer and we obtain near state-of-the-art results with BinaryConnect on the permutation-invariant MNIST, CIFAR-10 and SVHN.

This paper suggests to modification to the backpropagation algorithm which enables training of multilayer neural networks with binary weights. This is done by binarizing the weights only during the forward and backward propagation phases (while updating their real value stored). This should accelerate training speed, and much more significantly, testing speed.The fact that one can use binary weights during training, and still get good test performance, is novel and rather surprising (I had to check the code to believe this). The (minor) comments I have are about the relation to previous works: 1. It is not clear, why on line 373, it is said that the comparisons were done on different datasets. Indeed ref [29] is done for text classification, but doesn't ref [30] use the same regular (permutation invariant) mnist as in this paper? If that is the case, I see several possible explanations to the differences in results. First, EBP didn't use the tricks used in this paper (L2-SVM output, batch normalization, and adjustable/decaying learning rate), except dropout. Increasing the margin (using L2-svm) can be especially useful for decreasing the discretization sensitivity. Second, larger networks and many more epochs were used here. Third, in EBP also the neurons are binary (which is more hardware friendly then the hard sigmoid used here).2. Lines 369-371 are slightly inaccurate. In EBP [29,30]: it is also possible to use only a single binary network, at the price of reduced performance (as was done in [30]). This is somewhat similar to the decrease of performance in BinaryConnect (det.) over BinaryConnect (stoch.). Also, there a few typos and grammatical error ("Let us compared..."). In summary, very interesting paper!%%%Edit, after Author's Feedback: I raised the score from 8 to 9 due to the new experimental results, which seem quite promising. I mostly like very much the fact that you can train with binary weights, in contrast to previous works. Note that is FPGA section should be improved. It should be made more concrete (showing at least a diagram how the weights are placed and how the data are routed through the network. Specifically, how to route the convolutional layers) or removed.