Sparse and Low-Rank Tensor Decomposition
Motivated by the problem of robust factorization of a low-rank tensor, we study the question of sparse and low-rank tensor decomposition. We present an efficient computational algorithm that modifies Leurgans' algoirthm for tensor factorization. Our method relies on a reduction of the problem to sparse and low-rank matrix decomposition via the notion of tensor contraction. We use well-understood convex techniques for solving the reduced matrix sub-problem which then allows us to perform the full decomposition of the tensor. We delineate situations where the problem is recoverable and provide theoretical guarantees for our algorithm. We validate our algorithm with numerical experiments.

In this submission, the authors propose to solve a tensor factorisation problem of the form Z=X+Y, where X is a low-rank representation of Z and Y represents the sparse corruption. The authors define two subspaces of matrices T_{U,V} and T_{V,W} for a third-order non-symmetric tensor that describe the span of tensor after contraction along either the 1st or 3d mode, respectively. The sparsity after contraction along mode m is characterised by the union of support sets of each matrix (tensor slice) extracted along this mode. Therefore, the authors define incoherence parameters based on L_{inf} norms to control the sparsity, e.g. make spectra of matrices 'diffuse' rather than sparse. The authors employ an extension of the simultaneous diagonalisation [18] by contracting a tensor along modes 1 and 3 by vectors a and b randomly chosen for uniformly distributed unit sphere. This allows obtaining a set of eigenvalues u and v, and v' and w. After pairing eigenvalues u, v, and w, a set of equation arises in which the authors can solve for eigenvectors. For simultaneously imposing low-rank X and sparse Y, the authors apply the simultaneous diagonalisation on contractions of X and Y, respectively, and impose the nuclear norm on all X and \ell_1 norm on all Y, and solve equation 6. This particular subproblem is well explained in [10], e.g. equation 1.3.For evaluations, the authors generate 50^3 dimensional non-symmetric tensors Z composed from low rank X and sparse Y according to distributions described in the paper. The authors present the likelihood of recovery of tensors X and Y from Z w.r.t sparsity and true tensor rank r.  Pros: - an interesting non-symmetric tensor factorisation/decomposition problem - interesting expansion of existing symultaneous positive-definite matrix diagonalisation problem [18]. - interesting way of imposing the low-rank and sparsity related constraints on contractions extending problem [10]Cons: - this submission would benefit from more evaluations, e.g. beyond figure 1 - the last steps of the algorithm in section 2 could be described better - the proofs feel a little bit hard to follow. Major comments: 1. This is a very interesting paper that extends several ideas verging from simultaneous diagonalisation for a set of matrices to low-rank and sparse matrix recovery by applying/extending these methods to third order non-symmetric tensors. The maths behind these methods seems convincing.2. The symultaneous diagonalisation in [18] identifies the problem with using merely one projection vector (equvalent of one contraction). The authors of [18] note in section 3 that the quality of estimated eigenvalues depends on the value of the smallest eigengap. Hence, the authors of [18] propose random projections in order to minimise the situations where a single projection could lead to zeroing some eigenvalues. Do authors of this submission perform multiple contractions? It feels so however it is still very unclear form section 2 or algorithm 1. How does this affect the stability of diagonalisaton given non-symmetric tensors require solving for three sets of eigenvectors?3. The evaluations of the proposed algorithm feel somewhat limited. It'd be very convincing to see an experiment on non-synthetic data. Also, some of the guaranteesgiven in section 3 could be also illustrated in the experimental setting. For instance, what happens in practical simulations when v_3 and v_1 are outside of the guaranteed regime from corollary 3.4?This paper should be interesting to the community researching higher-order tensors. It is sufficiently clear and original despite it relies heavily on prior developments.


Covariance-Controlled Adaptive Langevin Thermostat for Large-Scale Bayesian Sampling
Monte Carlo sampling for Bayesian posterior inference is a common approach used in machine learning. The Markov Chain Monte Carlo procedures that are used are often discrete-time analogues of associated stochastic differential equations (SDEs). These SDEs are guaranteed to leave invariant the required posterior distribution. An area of current research addresses the computational benefits of stochastic gradient methods in this setting. Existing techniques rely on estimating the variance or covariance of the subsampling error, and typically assume constant variance. In this article, we propose a covariance-controlled adaptive Langevin thermostat that can effectively dissipate parameter-dependent noise while maintaining a desired target distribution. The proposed method achieves a substantial speedup over popular alternative schemes for large-scale machine learning applications.

Paper Title: Covariance-Controlled Adaptive Langevin Thermostat for Large-Scale Bayesian SamplingPaper Summary: This paper presents a new method (the "covariance-controlled adaptive Langevin thermostat") for MCMC posterior sampling for Bayesian inference. Along the lines of previous work in scalable MCMC, this is a stochastic gradient sampling method. The presented method aims to decrease parameter-dependent noise (in order to speed-up convergence to the given invariant distribution of the Markov chain, and generate beneficial samples more efficiently), while maintaining the desired invariant distribution of the Markov chain. Similar to existing stochastic gradient MCMC methods, this method aims to find use in large-scale machine learning settings (i.e. Bayesian inference with large numbers of observations). Experiments on three models (a normal-gamma model, Bayesian logistic regression, and a discriminative restricted Boltzmann machine) aim to show that the presented method performs better than stochastic gradient Hamiltonian monte carlo (SGHMC) and stochastic gradient Nose-Hoover thermostat (SGNHT), two similar existing methods. Comments:- I feel that this paper proposes a valid contribution to the area of stochastic gradient MCMC methods, and does a good job putting this method in context with similar previous methods (SGHMC and SGNHT). However, one detriment of this paper is that it is somewhat incremental, both in terms of ideas and the results shown.- In the experiments, comparisons are only against two methods: SGHMC and SGNHT. It might be nice to also see results for some of the other recently developed mini-batch MCMC methods, (such as the original stochastic gradient Langevin dynamics, or stochastic gradient Fisher scoring), or for some of the methods that do not rely on stochastic gradients, such as: (Bardenet, Remi, Arnaud Doucet, and Chris Holmes. "On Markov chain Monte Carlo methods for tall data." arXiv preprint arXiv:1505.02827 (2015)), or (Maclaurin, Dougal, and Ryan P. Adams. "Firefly Monte Carlo: Exact MCMC with subsets of data." arXiv preprint arXiv:1403.5693 (2014)).- In Figure 1, the inset "peaks" add very little to the figure -- they seem to be an only very-slight zoom into what is shown in the non-inset part of the figure.- I feel there are a few places in this paper where the quality of writing could be improved. In the abstract, there are few sentences that feel somewhat ambiguous to me (such as "one area of current research asks how to utilise the computational benefits of stochastic gradient methods in this setting."). In the intro (second paragraph), the order of presentation of stochastic gradient methods seems odd (first, the collection of all existing methods are described, and then afterwards, the first developed method is described). In section 2, there is a bit of confusion when a few terms are introduced without enough description (such as "temperature", "Boltzmann constant"); it would be better to give a brief description or intuition when introducing these terms to a machine learning audience.


Deep Poisson Factor Modeling
We propose a new deep architecture for topic modeling, based on Poisson Factor Analysis (PFA) modules. The model is composed of a Poisson distribution to model observed vectors of counts, as well as a deep hierarchy of hidden binary units. Rather than using logistic functions to characterize the probability that a latent binary unit is on, we employ a Bernoulli-Poisson link, which allows PFA modules to be used repeatedly in the deep architecture. We also describe an approach to build discriminative topic models, by adapting PFA modules. We derive efficient inference via MCMC and stochastic variational methods, that scale with the number of non-zeros in the data and binary units, yielding significant efficiency, relative to models based on logistic links. Experiments on several corpora demonstrate the advantages of our model when compared to related deep models.

This paper proposes a novel probabilistic deep architecture for modelling count data. In addition the authors propose a modelling extension for multi-class classification problems (i.e., discriminative modelling). The authors derive two inference techniques, one sampling one and a variational inference one, and show empirical gains compared to unsupervised methods using several large-scale datasets.Questions/Comments: - I found the model to be clearly explained and motivated. I enjoyed reading Section 2.3. I think adding a figure to illustrate Equations 3 would be useful. - I think it may be slightly misleading to imply that [23] does not scale wrt to the number of zeros. My understanding is that using a Poisson likelihood allows all methods to effectively ignore the zeros. Further it seems like most of the computational gains come from that rather than from higher-layers (which are typically much smaller). [23] also does allow  non-linear activation functions. - Computation. It appears that your model scales well. It would be interesting to have an idea of how computation scales and how long it takes to learn on these larger datasets. Providing a rough comparison to competing models (docNADE, LDA, and replicated softmax) would also be useful. - When reporting the results of the classification experiments it seems like you are only comparing to unsupervised techniques. In that sense the comparison is not absolutely fair. It would be good to add, at least, one simple supervised baseline (e.g., a small neural net with a softmax output and the word frequencies as inputs).Other comments: - line 165: Sparsity of the Dirichlet relies on your choice of parameters (eta). I think it would be good to make it clear.


Monotone k-Submodular Function Maximization with Size Constraints
A k-submodular function is a generalization of a submodular function, where the input consists of k disjoint subsets, instead of a single subset, of the domain. Many machine learning problems, including influence maximization with k kinds of topics and sensor placement with k kinds of sensors, can be naturally modeled as the problem of maximizing monotone k-submodular functions. In this paper, we give constant-factor approximation algorithms for maximizing monotone ksubmodular functions subject to several size constraints. The running time of our algorithms are almost linear in the domain size. We experimentally demonstrate that our algorithms outperform baseline algorithms in terms of the solution quality.

This paper studies the problem of maximizing monotone k-submodular maximization under either a total size constraint or a set of individual size constraints. For both classes of constraints it proposes two variants of greedy algorithms: k-greedy and k-stochastic greedy. For the total size constraint case, both algorithms, in general, provide a guarantee of 1/2. And a guarantee of 1/3 is given for the individual size constraint case. In the experiments, the proposed algorithms are observed to give better function valuation than baseline methods. This paper study the extension of submodularity to k-submodularity and generalizes the classical greedy algorithm to this setting. However, I think it would very interesting to see more applications of k-submodularity besides sensor placement and influence maximization. The stochastic variant of the greedy algorithm is very similar to the following paper:Lazier Than Lazy Greedy, Mirzasoleiman et al AAAI 2015.Instead of showing that the stochastic greedy gives a solution with bound 1/2 with probability 1-\delta, I think the authors can also give an expectation bound: the stochastic greedy algorithm always gives a solution on expectation with bound 1/2 - \delta. In the experiments, it seems that the performance of the stochastic greedy sometimes works even better than the greedy algorithm (as shown in Fig 1 and 3), which is different from the results as reported in the lazier than lazy greedy paper. I am hoping the authors could further investigate why and give some comments on this observation.   Minor comments:For the proposed algorithms, it seems that the authors actually have used the lazy evaluation trick (minoux' trick) in the experiments, but I think it would be better to describe and discuss how to use the lazy evaluation trick when describing the algorithms.line 60-62: One may not use the computationally expensive algorithm that is tight with bound (1-1/e) for solving submodular maximization under the partition matroid constraint. Instead, a simple greedy algorithm very much the same as the one proposed in this work can solve it with guarantee 1/2.  For the sensor placement experiment, it seems that the objective is defined as the entropy function. But it is unclear how the entropy function is evaluated, since this function requires exponential complexity to evaluate. Please describe in more detail.


Generative Image Modeling Using Spatial LSTMs
Modeling the distribution of natural images is challenging, partly because of strong statistical dependencies which can extend over hundreds of pixels. Recurrent neural networks have been successful in capturing long-range dependencies in a number of problems but only recently have found their way into generative image models. We here introduce a recurrent image model based on multidimensional long short-term memory units which are particularly suited for image modeling due to their spatial structure. Our model scales to images of arbitrary size and its likelihood is computationally tractable. We find that it outperforms the state of the art in quantitative comparisons on several image datasets and produces promising results when used for texture synthesis and inpainting.

The basic idea of this paper is to replace the MCGSM (mixture of conditional Gaussian scale mixtures) of [38] with a version where a continuous-valued hidden state vector h is maintained in a LSTM. This is used as a model of natural images and assessed by a density estimation task (secs 3.2 and 3.3, Tables 1-3), and for texture synthesis and inpainting (sec 3.4).The model for p(x_ij|h_ij) (l 150) is in fact not specified at all. Given that h is a continuous-valued vector (es per eq 6) we need to see some functional form. RNADE [41] is designed for fixed-length vectors. In the case of "infinite dimensional images" it would be very simple to build a mixture density network (MDN) or mixture of experts (see e.g. Bishop's 1995 book sec 6.4) to predict x_{ij} given a finite window of context (as per Fig 1A); (R)NADE is used in the more complex case of fixed-length data vectors. This comparison needs to be made to properly evaluate the benefit of RIDE in the Tables, as the MCGSM (as per eqs 4 and 5) is a relatively simple MDN. RIDE has theoretical benefits of potentially using infinite context as opposed to finite context, cf FIR vs IIR filters, but this theoretical benefit needs to be demonstrated in practice.RIDE is compared with variants of the MCGSM on "infinite dimensional images", and as per Tables 1 and 2 (RH column) it shows small advantages over MCGSM. However, as per the above para, we need to see comparisons with a MDN which is richer than a MCGSM (e.g. using hidden layers).The authors claim (l 371) that inpainting has only been done for small image regions until recently (citing [32] from 2015).However, there is work before this inpainting large regions, see e.g.Multiple Texture Boltzmann Machines, J. J. Kivinen and Christopher K. I. Williams, Proceedings AISTATS 2012, http://homepages.inf.ed.ac.uk/ckiw/postscript/mtbm.pdf . In that paper the authors consider the obvious evaluation of inpainting, computing the normalized cross correlation (NCC) of the inpainted region with the ground truth; this method should be used toevaluate the RIDE model quantitatively. Also in the same paper the authors used a texture similarity score (TSS) for assessing thequality of texture samples. Quality: The paper makes sense technically, using the spatial LSTM model of [8] to model images.Clarity: Writing is clear, except for defn of p(x_ij|h_ij) (l 150).Originality: A relatively straightforward application of the spatial LSTM framework (although getting it all to work well would require a lot of effort unless a good spatial LSTM code base is to hand).Significance: As per the comments above, we need to see (1) comparisonwith a MDN predictor (with careful effort to make it work as well as it can, as you have done for LSTM), and (2) better comparisons for texture synthesis/inpainting.Overall: Marginally below threshold due to lack of comparisons for density estimation and texture work.Other points:l 135 provide name for g units.l 280 Table 3 Aveage -> Averagel 312. The "dead leaves" model was introduced by Matheron in 1968, and further developed by D. Jeulin, see e.g.Jeulin, D. (1989). Morphological modeling of images by sequential random functions. Signal Processing 16, 403-431.Jeulin, D. (1996). Dead leaves models: From space tesselation to random functions. In Advances in Theory and Applications of Random Sets. ed. D. Jeulin. pp. 137-156.l 359 -- give url for where you obtained Brodatz data.ADDED AFTER REBUTTAL: =====================The other reviews and the author response have been read. The author clarification that the model for p(x_ij|h_ij) is a MCGSM means that the results are making a fair comparison between a MCGSM for p(x_ij|x_{< ij}) and p(x_ij|h_ij).I have thus raised my score to 6.I still believe that using a more complex model than a MCGSM (with hidden layers) may lead to benefits and encourage the authors topursue this line of enquiry.


Lifelong Learning with Non-i.i.d. Tasks
In this work we aim at extending the theoretical foundations of lifelong learning. Previous work analyzing this scenario is based on the assumption that learning tasks are sampled i.i.d. from a task environment or limited to strongly constrained data distributions. Instead, we study two scenarios when lifelong learning is possible, even though the observed tasks do not form an i.i.d. sample: first, when they are sampled from the same environment, but possibly with dependencies, and second, when the task environment is allowed to change over time in a consistent way. In the first case we prove a PAC-Bayesian theorem that can be seen as a direct generalization of the analogous previous result for the i.i.d. case. For the second scenario we propose to learn an inductive bias in form of a transfer procedure. We present a generalization bound and show on a toy example how it can be used to identify a beneficial transfer algorithm.

The paper presents some extensions to the Pentina and Lampert's PAC-Bayesian analysis of "Lifelong Learning" problems (ICML 2014) , where a learner must adapt to various tasks exploiting knowledge from previously seen ones. The main contributions are risk bounds dedicated to two scenarios where the observed task are not sampled independently from each other. Roughly speaking, the first scenario share similarities with domain adaptation (albeit the risk bound is given on an average of all possible domains, instead of on a specific target domain) and the second is quasi-identical (up to my knowledge) to distribution drift. In the first setting (Section 3), the authors cleverly reuse Ralaivola et al.'s chromatic PAC-Bayesian theory to represent dependencies between tasks. However, this result alone let me unsatisfied. I wonder to which extent this result can be useful to the ambitious "lifelong learning" problem the authors are interested in. At the end of Section 3, the authors open the way to use the result as a "quality measure" benchmark or to the conception of a learning algorithm. If provided, such empirical study could be a good way to enhance this work.The second setting (Section 4) is very similar to the "distribution drift" scenario. I would like the authors to compare their results with other ones in this area (e.g., Mohri and Medina's "New Analysis and Algorithm for Learning with Drifting Distributions", ALT 2012). I also think that this setting must be studied more deeply. Contrary to the author claim, I think that the assumption of Equation (18) is very restrictive: it is unlikely that *all* algorithms of a considered family would have *exactly* the same error on every two consecutive tasks. This assumption should be relaxed to become realistic (as in Mohri and Medina 2012). Moreover, the algorithm should be able to consider more than the two previous tasks.Finally, throughout the paper, I wondered why the authors choose this specific formulation of PAC-Bayesian theorems. While Theorem 1 tradeoff is very easy to understand, the obtained bound is less tight than the bound of McAllester ("PAC-Bayesian stochastic model selection", Mach Learn 2003), where the complexity term appears under a square root, while being "explicit" (see also the slightly tighter version of Germain et al.'s "Risk Bounds for the Majority Vote...", JMLR 2015). Consequently, I presume that the further bounds for non-i.i.d. tasks are not as tight as they can be. I think this must be discussed in the paper.Some typos and minor comments: - There is a small error in Theorem 1 proof: When applying Hoeffding's lemma (Equation (3) of the Supplementary Material), there should be a "4" before lambda^2, because the subtraction between losses lies in [-1,1]. I also think that the same error appears at Equations (11), (25) and (26) of the main paper. - Line 89: The dot is misplaced - Supplementary Material, Line 78: Fro -> For - For the reader's benefit, it is preferable to cite the more recent version of Ralaivola et al.'s "Chromatic PAC-Bayes bounds for non-iid data" (JMLR 2010 instead of AISTATS 2009) - I personally prefer when the definitions, theorems, lemmas, etc share the same counter.


Cornering Stationary and Restless Mixing Bandits with Remix-UCB
We study the restless bandit problem where arms are associated with stationary ϕ-mixing processes and where rewards are therefore dependent: the question that arises from this setting is that of carefully recovering some independence by 'ignoring' the values of some rewards. As we shall see, the bandit problem we tackle requires us to address the exploration/exploitation/independence trade-off, which we do by considering the idea of a waiting arm in the new Remix-UCB algorithm, a generalization of Improved-UCB for the problem at hand, that we introduce. We provide a regret analysis for this bandit strategy; two noticeable features of Remix-UCB are that i) it reduces to the regular Improved-UCB when the ϕ-mixing coefficients are all 0, i.e. when the i.i.d scenario is recovered, and ii) when ϕ(n) = O(n −α ), it is able to ensure a controlled regret of order Θ ∆ (α−2)/α * log 1/α T , where ∆ * encodes the distance between the best arm and the best suboptimal arm, even in the case when α < 1, i.e. the case when the ϕ-mixing coefficients are not summable.

Summary: The paper studies the problem of restless mixing bandits, more specifically the case when rewards of an arm evolve according to a \phi-mixing process. In this problem the rewards distribution of an arm is not stationary and evolves according to a \phi-mixing process where the correlation between rewards of an arm in two different rounds decrease over time. The paper proposes an algorithm called Remix-UCB (a variant of Improved-UCB for the stationary case). In this algorithm we maintain confidence bounds on rewards of each arm. Then we proceed in epochs and play each arm certain number of times before eliminating clearly suboptimal arm. The key new idea is that at the end of each of these epochs the algorithm decides to play the best arm certain number of rounds and not use information in these rounds to update confidence bounds. The paper shows regret bound of \theta( (\log T)^{1/\alpha} \delta^(\alpha-1)/alpha) with respect to bench mark of single best arm. Clarity: The paper is written in a reasonably clear manner. Significance of the problem: Restless bandits are an important class of bandit problems which model many interesting problems in machine learning on online problems. Any good result should be of interest to the community. Originality: Although the paper studies the classic problem of restless ucb, the specific variant has not been studied in the literature. This variant looks quite natural and should be of interest. The algorithm that the paper proposes is quite natural and borrows heavily on several techniques in the area including using confidence bounds to estimate mean reward and iterated prunning of suboptimal arms. But the paper also introduces a new idea of playing the best known arm without updating confidence bounds in their algorithm which makes sense in practice. Comments: The focus of the paper is on getting asymptotic regret which scales like a power of \log(T). If we wanted something of \sqrt{T} regret we can just use exp3 algorithm as the benchmark is single best arm. Main issues: - One of the interesting aspects of restless bandits is the fact that we can define stronger benchmarks which can play different arms in each round. So the fact that the paper competes against single best arm is a bit weak. - While it is true that we tend to ignore constant in regret bounds the paper is a bit too loose in regret analysis. It does not show dependence on parameters such as k.- technical correction \lambda_k^\tau = O(t^{3/4}) (line 236 and similarly line 262) - Improved UCB gives interesting bounds when \sum \phi < O(\sqrt{t}) and not just as \sum \phi < constant as the paper indicates.


Parallel Predictive Entropy Search for Batch Global Optimization of Expensive Objective Functions
We develop parallel predictive entropy search (PPES), a novel algorithm for Bayesian optimization of expensive black-box objective functions. At each iteration, PPES aims to select a batch of points which will maximize the information gain about the global maximizer of the objective. Well known strategies exist for suggesting a single evaluation point based on previous observations, while far fewer are known for selecting batches of points to evaluate in parallel. The few batch selection schemes that have been studied all resort to greedy methods to compute an optimal batch. To the best of our knowledge, PPES is the first nongreedy batch Bayesian optimization strategy. We demonstrate the benefit of this approach in optimization performance on both synthetic and real world applications, including problems in machine learning, rocket science and robotics.

Paper Summary: Motivated by the optimization of objective functions which are extremely expensive to evaluate, but which can be accessed in parallel, the authors construct an algorithm for selecting batches of several instances to evaluate, called Parallel Predictive Entropy Search, or PPES.PPES is an extension of the Predictive Entropy Search (PES) algorithm of Hernandez-Lobato et al., 2014.PPES is constructed such that the instances which compose a batch jointly (i.e., non-greedily) optimize an approximate expected conditional mutual information between the corresponding observations and the location of the global optimizer of the objective function. In order to accomplish this, the authors develop an approximation to the mutual information using EP, construct an approximate sampler from the posterior over the global maximizer's location, and address how the optimization over the approximate expected conditional mutual information (i.e., the selection of the instances for the batch) can be performed.They then proceed to test their approximate expected conditional mutual information in a simple setting.Finally, they test the whole algorithm on a variety of data sets against several competing batch algorithms.Quality: The approach taken by the authors is rigorous, sensible, and produces useful results.There are several approximations, but they are well-motivated.The numerical experiments are quite complete, though they could do with a bit more information in the supplement as to precisely what code was being run.The approximation which most interests me is the one taken in Equation 4; essentially, the authors approximate p(f|D,S_t,x*) by a PDF conditioned on a series of necessary (f(x) < f(x*)) or quasi-necessary (f(x*) >= ymax + eps) conditions; how does this affect the entropy of the resulting distribution?Intuitively, I would expect that conditioning only on necessary conditions would produce an entropy which is at least as large as the original, but what about the quasi-necessary conditioning?Should we expect the information gain to be consistently under-estimated?Is it reasonable to expect that the quasi-necessary condition won't bias the location of the maximizer?Some intuition from the authors might be helpful here and in other places. Clarity: The material and algorithm are both quite complicated, but the presentation is well done and this explanation is fairly clear.Section 3, the major technical section, is particularly well constructed.  In regard to the experiments, it would be best if the implementations of the competing algorithms were discussed a bit, perhaps in the supplementary material; some of the algorithms are not explicitly designed to give a most probable maximizer, and the code for others (GP-BUCB, at least, and perhaps the Azimi et al. simulation matching algorithms) are unfortunately publicly available only in a form which takes discrete input sets; without having this information in some form, it's a bit hard to assess how much value the experiments have.PPES looks to perform quite well, clearly, but how strong are the baselines?Of course, the authors of this work should not be responsible for fixing the code of others, but some information on this would be handy.   Originality: The primary basis for the paper is Hernandez-Lobato, Hoffman, and Ghahramani 2014.Relative to that work, which introduced PES, the PPES algorithm is a parallelized version that allows the selection of batches of actions.Practically, this is an important new capability.So far as I can tell, the primary theoretical differences are the removal of a condition from the approximation of the predictive entropy (conditioned on x*) in section 3.1, which appear to make this computation similar, and some differences in computing the gradient of the approximate decision function in section 3.3.The EP implementation may also be substantially different, but this is somewhat hard to tell from the text.These innovations do not appear to be enormous, or fundamental changes of approach.Undoubtedly, there was plenty of work which went into extending PES to PPES, but the authors have not done a good job of highlighting precisely what the theoretical novelties of the present work are, or why they were necessary or interesting.  Significance: This work appears to be a good fit for the setting of controlling computationally intensive simulations, and will likely make a good impact there.In particular, in this setting the method's high computational cost would be still very small as compared to the large simulations it would be controlling, the parallelism it presumes is available, and the experiments are very expensive.I would argue that this combination of strong writing, good mathematical meat, and the availability of a well-suited problem domain make this a significant paper for practice. That said, I can imagine that this setting would also often have batch sizes which are larger than the 2-4 attempted in this paper, and as the authors point out, their algorithm scales poorly for these cases.How does this cost compare against the computational expense of a single observation in a reasonable setting, e.g., the robot problem addressed here? At what point is it reasonable to run a simpler algorithm to allow greater parallelism?Again, some practical guidance would be helpful.


Exactness of Approximate MAP Inference in Continuous MRFs
Computing the MAP assignment in graphical models is generally intractable. As a result, for discrete graphical models, the MAP problem is often approximated using linear programming relaxations. Much research has focused on characterizing when these LP relaxations are tight, and while they are relatively well-understood in the discrete case, only a few results are known for their continuous analog. In this work, we use graph covers to provide necessary and sufficient conditions for continuous MAP relaxations to be tight. We use this characterization to give simple proofs that the relaxation is tight for log-concave decomposable and logsupermodular decomposable models. We conclude by exploring the relationship between these two seemingly distinct classes of functions and providing specific conditions under which the MAP relaxation can and cannot be tight.

This paper provides a theoretical study of tightness of the LP relaxations for MRFs with continuous variables. The paper provides the following contributions: - Generalization of the results of [13] to the case of continuous MRFs: formulation of the value of the LP relaxation via graph covers (theorem 3.1) - Formulation of the two special cases when the LP relaxation is tight (log-concave decomposable and log-supermodular decomposable models). Proofs are based on theorem 3.1. The paper introduces the family of log-supermodular decomposable functions. - Condition allowing to identify cases when the MAP relaxation is not tight (theorem 4.7). The paper is relatively well written, although some parts are quite hard to follow. The derivations presented in the paper seem to be correct. I have several comments related to correctness/clarity: 1) Proof of theorem 3.3 is based on the repeated application of the definition of log-supermodularity (line 222) to the l.h.s. of line 233. This definition allows to upper bound the product of function values at different point by product of the function values at the point-wise maximum and minimum. At the same time the r.h.s line 233 contains only point-wise maximum (z^m) and not the point-wise minimums. Is this error critical for the result to hold? 2) Section 4.1 should be explaining relation of the two families of the functions in case of Gaussian MRFs. I could not get the relation based on the provided explanation. Lines 280-282 were not enough for my understanding. 3) I did not understand why section 4.2 was included into section 4, because it seemed to provide a separate result. The presented results seem to be original. The connections to other existing results are explained although sometimes are not reader-friendly (e.g. lines 354-355).


Stochastic Expectation Propagation
Expectation propagation (EP) is a deterministic approximation algorithm that is often used to perform approximate Bayesian parameter learning. EP approximates the full intractable posterior distribution through a set of local approximations that are iteratively refined for each datapoint. EP can offer analytic and computational advantages over other approximations, such as Variational Inference (VI), and is the method of choice for a number of models. The local nature of EP appears to make it an ideal candidate for performing Bayesian learning on large models in large-scale dataset settings. However, EP has a crucial limitation in this context: the number of approximating factors needs to increase with the number of datapoints, N , which often entails a prohibitively large memory overhead. This paper presents an extension to EP, called stochastic expectation propagation (SEP), that maintains a global posterior approximation (like VI) but updates it in a local way (like EP). Experiments on a number of canonical learning problems using synthetic and real-world datasets indicate that SEP performs almost as well as full EP, but reduces the memory consumption by a factor of N . SEP is therefore ideally suited to performing approximate Bayesian learning in the large model, large dataset setting.

QUALITY This is an interesting idea. However, I think the paper requires two major changes to be interesting to a NIPS reader.First, the paper should be up front about its experimental nature and follow with an in-depth study. Specifically, the experiments should clearly highlight the strengths (big datasets) and weaknesses (clear, thorough comparisons and failure cases).Second, the paper requires some editorial work. The important sections (see below) could use more explanation. The experimental study should clearly motivate the setup and carefully discuss the results with technical detail. This is what a NIPS reader would want to get from such a paper.Some details:Bayesian Probit Regression: I do not understand why this approximate KL is a good metric to report. Why not just compare means and variances of parameter estimates as compared to Stan/NUTS? Isn't that much more straightforward and natural?Figure 3(c): I am confused as to why SEP with K=1 is compared to DSEP with K=10. Am I missing something here?Mixture of Gaussians: why did you now switch to this F-norm metric? Also wouldn't it be better to simulate some data where EP fails to recover the truth? I would rather want to see whether SEP "fails in the same way" as EP, or whether the single f(\theta) factor provides a different behavior.Having reached Section 5.3, I am a bit dissatisfied by what these first two experimental sections present. I have not learned much about the nature of SEP: specifically, I don't understand how it fails. The investigation of minibatch sizes is limited, as the experiments do not paint a clear picture. Also, given that the main motivation of this work is to scale EP, I don't understand why the majority of the experiments focus on tiny datasets.Section 5.3 requires significant overhaul. The experimental setup is brushed aside with a citation: this is unacceptable. The results are discussed in a completely conversational manner, with little technical detail or specificity, using words like "often", "interestingly", "possibly", and "likely".CLARITY Figure 3(b): DAEP is never discussed, either in the caption or in the text.I am not sure what Figure 2 adds to the paper. I would rather the authors expand on Sections 3 and 4.1: both of these sections are important for the paper, and yet they would benefit from a clearer exposition. Removing Figure 2 would afford this space to clarify these important ideas of the paper.ORIGINALITY The idea is nice. It is also original: i have not seen any work that addresses EP's memory constraints. The proposal is straightforward; thus a NIPS reader will immediately ask: "that's a good idea, I wonder why it works."SPECIFICS line 48: ... having A myriad OF local ...line 69: truelyline 149: summariedline 199: arguableline 214: this paragraph is vague. I cannot tell what you are referring to without looking at the supplementary material; thus I have little motivation to look it up in the supplementary material.line 257: why is "mixture" capitalized?figure 2 should appear on page 5, not page 6.line 290: i am thoroughly confused by this comparison metric.line 292: there is no citation numbered [25]. should it be [24]? This makes me worried about all of the citations in the rest of the paper...


Efficient Output Kernel Learning for Multiple Tasks
The paradigm of multi-task learning is that one can achieve better generalization by learning tasks jointly and thus exploiting the similarity between the tasks rather than learning them independently of each other. While previously the relationship between tasks had to be user-defined in the form of an output kernel, recent approaches jointly learn the tasks and the output kernel. As the output kernel is a positive semidefinite matrix, the resulting optimization problems are not scalable in the number of tasks as an eigendecomposition is required in each step. Using the theory of positive semidefinite kernels we show in this paper that for a certain class of regularizers on the output kernel, the constraint of being positive semidefinite can be dropped as it is automatically satisfied for the relaxed problem. This leads to an unconstrained dual problem which can be solved efficiently. Experiments on several multi-task and multi-class data sets illustrate the efficacy of our approach in terms of computational efficiency as well as generalization performance.

The present paper focus on kernel-based multi-task learning. More specifically, it deals with learning the output kernel which defined over multiple tasks. Starting from the vector-valued RKHS formulation of the multi-task learning problem, the authors propose an output kernel learning algorithm that are able to learn both the multi-task learning function and task dependencies. The authors first provide an optimization formulation of the problem which is jointly convex. Then they show that this optimization can be solved without the positive-semidefiniteness constraintof the output kernel, which is computationally costly, and propose the use of a stochastic dual coordinate ascent method to solve it.Experiments on multi-task data sets and comparison in terms of performance and running time with previous multi-task learning algorithms are provided. The paper builds upon recent studies on multitask learning and output kernel learning. In particular, it is closely related to: [1] Dinuzzo & al., Learning output kernels with block coordinate descent., ICML 2011. [2] Zhang & Yeung, A convex formulation for learning task relationships in multi-task learning, UAI 2010.While these two references are clearly mentioned in the paper, there are other closely related papers which are not discussed and cited. For example [3] Kadri & al., Multiple operator-valued kernel learning, NIPS 2012. [4] Scalable matrix-valued kernel learning for high-dimensional nonlinear multivariate regression and Granger causality, UAI 2013. [5] Ciliberto & al., Convex learning of multiple tasks and their structure, ICML 2015.These papers, in particular [4,5], needs to be cited and discussed. The proposed approach needs to be compared to them from a theoretical and experimental perspectives.The contribution is somewhat limited. The main novelty is the dual formulation of the optimization problem which does not need the positive-semidefiniteness constraint of the output kernel. However, it should be noted that even in the previous works [1] and [2] where an alternating optimization scheme is adopted, positive-semidefiniteness constraint is not needed. Incorporating the solution of the minimization over the multi-task learning function in the optimization problem of the output kernel learning (without the positive-semidefiniteness constraint) provides a positive semi-definite matrix over tasks. This is clearly mentioned in [1] (see Eq. 7) and [2]. The way this paper is written gives the impression that it is the first that avoid the optimization over S_+^\top.Also, the result about the joint convexity of the output kernel learning optimization problem is not new. This is largely discussed in [5] and also in [2]. It is true that [5] is a newly published paper but the contribution of the present workwill be more visible by taking into account recent results in this field. In the experiments, it is not clear why the proposed approach is compared to OKL in terms of running time and not in terms of performance. The paper is about output kernel learning, so the performance of the proposed method needs to be compared to those of [1, 4, 5].update: The authors did a good job in answering my questions. My main concern is about the presentation of the results and their novelty. I think that is technical novelty and originality in the proposed approach, however the authors should better explain this and make it clear what the novel contributions of the paper are in comparison to what is known.


Quartz: Randomized Dual Coordinate Ascent with Arbitrary Sampling
We study the problem of minimizing the average of a large number of smooth convex functions penalized with a strongly convex regularizer. We propose and analyze a novel primal-dual method (Quartz) which at every iteration samples and updates a random subset of the dual variables, chosen according to an arbitrary distribution. In contrast to typical analysis, we directly bound the decrease of the primal-dual error (in expectation), without the need to first analyze the dual error. Depending on the choice of the sampling, we obtain efficient serial and mini-batch variants of the method. In the serial case, our bounds match the best known bounds for SDCA (both with uniform and importance sampling). With standard mini-batching, our bounds predict initial data-independent speedup as well as additional data-driven speedup which depends on spectral and sparsity properties of the data.

The paper presents a variant of dual coordinate ascent for regularized empirical risk minimization with smooth convex loss functions and strongly convex regularizers. The authors derive an update scheme where in each iteration only a subset of dual coefficients are updated. The authors proof the convergence of their method for any sampling scheme producing those subsets. Moreover, they derive sampling schemes to speedup the convergence. In experiments, the authors verify their theoretical findings.The paper is well-written and scientific-sounding. The theoretical as well as the experimental results suggest that the proposed method outperforms existing algorithms when trained highly parallel, especially for very sparse data. My only concern is about the experiments which are not very conclusive: - combination of dataset, setting and method is not consistent - why not performing serial and mini-batch experiments for all 6 datasets? - why not reporting SPDC-roh=1 together with the other methods the serial setting? - likewise, why not reporting on ASDCA in the mini-batch setting? - no reporting of wall-clock time; and no reporting of predictive performance and/or values of the objective; it's hard to tell wether primal dual gaps of 1e-4 and 1e-10 make any difference in practise - no setting other than block size m (=1), loss (smoothed hinge), and regularizer (l2); consider m>1 where sparsity may significantly drop, Ridge regression where the Lipschitz constant is unbounded, etc. - only one large real-world dataset; consider rcv1 and/or imagenet - please detail on the choice of lambda. It has a significant impact on the convergence speed, but also on the predictive performance of the resulting model. For very large datasets, "optimal" lambda effectively approaches zero; which (practical) impact would this have on the convergence?Minor comments for the authors: - line 224: consider "the number of matrices A_i with any nonzero entry in row j" (it wasn't quit clear to me what "blocks" refers to) - Eq 17: colon - line 295: should be omega_j; consider adding max_j omega_j \leq n - "average sparsity", in line 295, is misleading; omega_j tilde is the "average number of non-zero entries for dimension j"; omega is an (absolute) measure of density rather than a (relative) measure of sparsity. Similarly, in Table 3, where 100% sparsity denotes a fully dense dataset. - titles in Figure 1 and, in particular, in Figure 2 are not very helpful; consider adding the 3 artificial datasets in Table 3; currently the sparsity (their key property) is only stated in the caption of Fig 2. - Figure 1: consider reordering the methods where both Quartz variants should be red - supp.: last line before Ch. 8 should be m > 1


Efficient Learning of Continuous-Time Hidden Markov Models for Disease Progression
The Continuous-Time Hidden Markov Model (CT-HMM) is an attractive approach to modeling disease progression due to its ability to describe noisy observations arriving irregularly in time. However, the lack of an efficient parameter learning algorithm for CT-HMM restricts its use to very small models or requires unrealistic constraints on the state transitions. In this paper, we present the first complete characterization of efficient EM-based learning methods for CT-HMM models. We demonstrate that the learning problem consists of two challenges: the estimation of posterior state probabilities and the computation of end-state conditioned statistics. We solve the first challenge by reformulating the estimation problem in terms of an equivalent discrete time-inhomogeneous hidden Markov model. The second challenge is addressed by adapting three approaches from the continuous time Markov chain literature to the CT-HMM domain. We demonstrate the use of CT-HMMs with more than 100 states to visualize and predict disease progression using a glaucoma dataset and an Alzheimer's disease dataset.

The paper presents new methods to learn parameters and then perform inference (either Viterbi or forward/backward) for continous time HMM models (CT-HMM) with non uniformly distributed time points. They solve the inference problem for the transition rate matrix Q by constructing a discrete time inhomogeneous HMM then applying two recently developed methods to get sufficient statistics for their EM (Expfm, Unif - from Hobolth & Jensen 2011). They follow with a detailed time complexity analysis for the different flavors of the their algorithm (Table 1) before proceeding to experimental evaluation. Experiments include a 5 state synthetic dataset and two disease progression datasets for Glaucoma and AD.Pros: The paper is mostly well written and the problem addressed is interesting in terms of the computational challenge, the modeling aspects, and the biomedical domain. The authors provide mathematical foundation for their approach building on previous work byHobolth & Jensen 2011 for the specific task at hand. The authors do a good job explaining the biomedical datasets/questions and relating their results to what is known about the disease.Cons: 1. The synthetic data evaluation is not sufficient. Why 5 states? 10^6 samples seems an awful lot of samples to train on an not representative of real life problems. The difference between states emission (line 319-320) seems too easy? That may explain why all their algorithm's variants seem to preform similarly. Harder settings and varying different parameters may flush out differences better and give a bette sense of how the algorithm can perform on real life data. 2. Sec. 5.2 is rather cryptic, especially the description of the competing methods. Consider adding details about those in the appendix to make this section more clear.Minor comments: 1. Fig2 is not self contained. Referring to it first (line 40) is premature. 2. Line 91 "the rate the process" 3. Line 161 "in only at" 4. Sec 5.2: the claim about binary search (lines 348-350) is not clear. 5. Line 399 "to shown"


Multi-Layer Feature Reduction for Tree Structured Group Lasso via Hierarchical Projection
Tree structured group Lasso (TGL) is a powerful technique in uncovering the tree structured sparsity over the features, where each node encodes a group of features. It has been applied successfully in many real-world applications. However, with extremely large feature dimensions, solving TGL remains a significant challenge due to its highly complicated regularizer. In this paper, we propose a novel MultiLayer Feature reduction method (MLFre) to quickly identify the inactive nodes (the groups of features with zero coefficients in the solution) hierarchically in a top-down fashion, which are guaranteed to be irrelevant to the response. Thus, we can remove the detected nodes from the optimization without sacrificing accuracy. The major challenge in developing such testing rules is due to the overlaps between the parents and their children nodes. By a novel hierarchical projection algorithm, MLFre is able to test the nodes independently from any of their ancestor nodes. Moreover, we can integrate MLFre-that has a low computational cost-with any existing solvers. Experiments on both synthetic and real data sets demonstrate that the speedup gained by MLFre can be orders of magnitude.

The authors present a node screening method for tree-based regularization which identifies inactive nodes (i.e. nodes the features of which have all zero coefficients) in a top down manner given the tree that defines the groups of variables. As usual the idea is to work with the sub-gradient of the regularizer, if the subgradient with respect to a node is in the strict set of subgradient at 0, then this node is set to zero. The problem in tree based group lasso is that it is hard to access the subgradient of each node, since we only have access at the mixture of the subgradients with respect to all nodes. The authors provide a method to project the mixture of subgradients at each node and examine if the subgradient falls in the region which give us sparsity, if yes then by eliminating that corresponding node we remove those inactive features.The proposed method makes the use of tree structured lasso on datasets with tens of thousands, or even hunderds of thousands of attributes.  The authors start by deriving the dual of the tree group lasso (which is equivalent to a projection problem on the feasible set), with dual variable $\vect \theta$ and feasible set F={$X^T \theta$ in the subdifferential at zero of the tree norm}. They start with the observation that some $\theta$ will be a part of the feasible set if $X^T\theta$ is on the feasible set and thus its projection on the feasible set is $X^\theta$ and they provide an hierarchical algorithm that given a vector $z$ projects it on the feasible set. The algorithm works by hierarchically decomposing the projection on the subdifferential at zero to a projection on the sum of two sets: the bounded norm ball set B and the sum of the sets of the bounded norm balls of all of each children nodes. The sizes of the norm balls are given by the importance, w^i_j, of the respective nodes, G^i_j. Using the hierarchical projection algorithm they can decompose $X^T\theta^*$ at the optimal solution as a sum of vectors $\ksi_j^i$ where each $\ksi_j^i$ is in the sudifferential set of tree norm with respect to the G^i_j node of the tree at the optimal solution of the primal $\beta^*$. Thus if one knows $X^T\theta^*$ then one can compute the inactive nodes, G_j^i, by looking at the $\ksi^i_j$ that are within the respective bounded radius ball. However since $\theta^*$ is not known the authors define a set that contains the optimal solution and solve for an upper bounding problem, providing along the way a close-form solution for the non-convex supremum problem.The screening rule takes as input a sequence of hyperparameters $\lambda$ and for each sub-sequent pair outputs the corresponding inactive nodes of the tree, i.e. thoese the supremum problem of which is smaller than the radius of the ball of the respective node.  The authors provide a small set of experiments on one artificial and one real-world dataset in which the method consistently offers a speedup of around 40 times. The paper is well written but not easy to follow, especially for someone who is not well-versed into optimization namely because it is quite technical consisting of a series of lemmas and theorems (just the appendix with the respective proofs is more than 20 pages).Given the amount of the material and the NIPS space limitations I am not sure it would have been possible, even though desirable, to give a more high level picture/intuitive picture of the basic components of the algorithm before going into the technical details. There seems to be a problem in the proof of Theorem 7. The definition of lambda max is in contradiction to ii) in that theorem, i.e. \lambda_max is the largest value that keeps y/\lambda in the feasible set, however in ii) we have y/\lambda \in F is equivalent to \lambda \geq \lamnbda_max. Page 5 column 218 what is the \Ksi^i_j set?On R1,2^* since these are supremums how often do we expect to see sup'>'w_j^i while the node is in fact inactive? any insights from the theory or experimental results?


Attractor Network Dynamics Enable Preplay and Rapid Path Planning in Maze-like Environments
Rodents navigating in a well-known environment can rapidly learn and revisit observed reward locations, often after a single trial. While the mechanism for rapid path planning is unknown, the CA3 region in the hippocampus plays an important role, and emerging evidence suggests that place cell activity during hippocampal "preplay" periods may trace out future goal-directed trajectories. Here, we show how a particular mapping of space allows for the immediate generation of trajectories between arbitrary start and goal locations in an environment, based only on the mapped representation of the goal. We show that this representation can be implemented in a neural attractor network model, resulting in bump-like activity profiles resembling those of the CA3 region of hippocampus. Neurons tend to locally excite neurons with similar place field centers, while inhibiting other neurons with distant place field centers, such that stable bumps of activity can form at arbitrary locations in the environment. The network is initialized to represent a point in the environment, then weakly stimulated with an input corresponding to an arbitrary goal location. We show that the resulting activity can be interpreted as a gradient ascent on the value function induced by a reward at the goal location. Indeed, in networks with large place fields, we show that the network properties cause the bump to move smoothly from its initial location to the goal, around obstacles or walls. Our results illustrate that an attractor network with hippocampal-like attributes may be important for rapid path planning.

The proposed method to model preplay and rapid path planning using successor representation is a very interesting and potentially important development. It is also in my opinion a more realistic implementation of model-based RL than tree search-based alternatives.However, a few important questions remain: - is computation of advanced quantities as described in sections 2 and 3 biologically realistic - not just from the perspective of the resulting output and dynamics, but the mechanistic implementation (e.g. how weights that are derived here using least-squares learned)? - how does such successor-based representation work in more realistic scenarios when familiarity with the environment is partial and how does it change with learning - can it reproduce typically observed behavioral learning curves? - can it also be applied to other domains than space, e.g. time cells? I'm aware that addressing these questions in detail may be out of the scope of this paper, but it would be helpful if these aspects could at least be discussed briefly.Clarity of the paper, particularly of the mathematical expressions and derivations, could be improved, e.g. line 2 of eq.3 or why a weak stimulus corresponds to alpha = epsilon = 0.05 explained in more detail.


Adaptive Primal-Dual Splitting Methods for Statistical Learning and Image Processing
The alternating direction method of multipliers (ADMM) is an important tool for solving complex optimization problems, but it involves minimization sub-steps that are often difficult to solve efficiently. The Primal-Dual Hybrid Gradient (PDHG) method is a powerful alternative that often has simpler sub-steps than ADMM, thus producing lower complexity solvers. Despite the flexibility of this method, PDHG is often impractical because it requires the careful choice of multiple stepsize parameters. There is often no intuitive way to choose these parameters to maximize efficiency, or even achieve convergence. We propose self-adaptive stepsize rules that automatically tune PDHG parameters for optimal convergence. We rigorously analyze our methods, and identify convergence rates. Numerical experiments show that adaptive PDHG has strong advantages over non-adaptive methods in terms of both efficiency and simplicity for the user.

Augmented Lagrangian algorithms like ADMM have become popular recently since they decouple an objective function consisting of the sum of several functionals. PDHG is a version were the subproblems have a particularly simple form, more specifically, they are proximal problems w.r.t. the individual functionals of the objective function. In this paper, the authors propose an adaptive step selection rule for PDHG, provide convergence guarantees and show the numerical performance of the new algorithm.The proposed algorithm seems to converge quickly according to the numerical experiments. However, there exist several PDHG algorithms which also use dynamic step sizes and have shown to outperform the methods used for comparison in the present paper. There is Algorithm 2 of [5] and the algorithm in "Adaptive Primal-Dual Hybrid Gradient Methods for Saddle-Point Problems" by T. Goldstein et al. (2013). The latter is particularly relevant here since it uses a very similar idea namely to balance the primal and dual residual. Without these comparisons it is a bit hard to judge how good the proposed algorithm actually is. A minor issue in this context is the question whether there are better constant step sizes than the two used in the experiments.Another very relevant paper which is not cited here is "On the convergence of primal-dual hybrid gradient algorithms for total variation image restoration" by S. Bonettini and V. Ruggiero (2012) since it shows convergence of PDHG for dynamic step sizes.Moreover, it would be useful to give a brief explanation of the result of Theorem 1. For example, one could clearly state that the nominator is bounded from above since u^k is bounded (Lemma 2). Also, providing the convergence result in terms of an average over {u^1,...,u^t} seems a bit unusual. Related to this is the question whether the sequences x^k or y^k converge. Minor issues: * It should be \tilde{u}^T in Theorem 1. * C_\phi in Lemma 3 should be defined before it is used. * In (20), \delta_k-1 is not defined. One could directly replace (1-\delta_k-1) with \phi_k-1


Teaching Machines to Read and Comprehend
Teaching machines to read natural language documents remains an elusive challenge. Machine reading systems can be tested on their ability to answer questions posed on the contents of documents that they have seen, but until now large scale training and test datasets have been missing for this type of evaluation. In this work we define a new methodology that resolves this bottleneck and provides large scale supervised reading comprehension data. This allows us to develop a class of attention based deep neural networks that learn to read real documents and answer complex questions with minimal prior knowledge of language structure.

This paper focuses on the reading comprehension task. It proposes to construct a large dataset using news stories and highlights from CNN and Daily Mail. Reading comprehension is formulated as a Cloze test, where a named entity in the news highlight is replaced by a placeholder. The system is expected to predict the missing entity (answer) based on the entire news document (document) and highlight (query). The paper explores several approaches to address the problem. The baselines are based on entity frequency, frame semantic parsing, and word distance. A set of neural models, including a multi-layer LSTM and LSTMs with attention mechanism, are compared against the baselines. Results show that adding the attention mechanism helps, and the 'impatient reader' model yields the best performance.The paper is well structured in general. The proposed approach for constructing supervised reading comprehension dataset using news documents and highlights seems interesting. I have a few concerns with the paper, they are listed below.It seems the baselines used in the paper are pretty weak. The frame semantic parsing and word distance benchmarks are both unsupervised and they seem to be ad-hoc. Although the frame semantic parser is used to retrieve predicate/argument triples, only heuristic rules are used to combine the triples for answer prediction. I feel the state-of-the-art NLP approaches could do much better than that. Perhaps even a supervised extension would generate better results.I'd also be curious to know the difficulty of the proposed Cloze test. In particular, how well human subjects would perform on the anonymized test. I suspect entities in the news highlight occur frequently in the document. It would be helpful to provide some coverage statistics. For example, the percentage of gold-standard answers that are covered in the top-3 or top-5 most frequent entities. If the coverage is high, perhaps the task is not as difficult as it may seem.Albeit the good performance, I'm not fully convinced that LSTM would be the method-of-choicefor this task. It seems adding the attention mechanism certainly works, since it helps identify the passages that are related to the answer. However, I'm a bit surprised to see that without the attention mechanism, the single-layer LSTM model yields much worse performance ('uniform attention' in Table 5). I'm also not sure why 'impatient reader' performs well. Some explanations on the advantage of this model would be helpful.The question categorization in Section 2.1 seems unclear. The definition of different categories needs clarification. Providing example queries may help. If space is a concern, it seems okay to completely remove this section.


Skip-Thought Vectors
We describe an approach for unsupervised learning of a generic, distributed sentence encoder. Using the continuity of text from books, we train an encoderdecoder model that tries to reconstruct the surrounding sentences of an encoded passage. Sentences that share semantic and syntactic properties are thus mapped to similar vector representations. We next introduce a simple vocabulary expansion method to encode words that were not seen as part of training, allowing us to expand our vocabulary to a million words. After training our model, we extract and evaluate our vectors with linear models on 8 tasks: semantic relatedness, paraphrase detection, image-sentence ranking, question-type classification and 4 benchmark sentiment and subjectivity datasets. The end result is an off-the-shelf encoder that can produce highly generic sentence representations that are robust and perform well in practice.

TL; DR This paper presents a natural generalization of neural word embeddings: sentence embeddings! The training objective of the proposed "skip-thoughts" model is to predict the previous and next sentence given an encoding of the current sentence. Thus, it may be trained without supervision from raw text, and the resulting embeddings used as "features" for downstream tasks. The authors present experiments on 8 tasks, and while the results of the proposed method are not state-of-the-art, it outperforms many previous baselines with less feature engineering.The proposed "skip-thoughts" model is a fairly ingenious application of the recently popularized sequence-to-sequence RNN framework. It's been known for a long time now (e.g., Ando & Zhang, 2005) that learning representations of observed data X via auxiliary tasks can be helpful in downstream tasks, in which one is interested in making predictions Y. However, doing so at the sentence level is a nice twist, though not an entirely novel one (e.g. the cited paragraph vector work, and earlier work cited therein). I think the technical approach in the paper, being largely derived from existing RNN frameworks, is sensible. I also liked the vocabulary expansion strategy.I do have some concerns regarding the experimental evaluation. The main question in my mind is if the proposed approach is a better way of learning from a big pile of unlabeled data X than alternatives, in terms of downstream performance on predicting Y. Unfortunately, I don't believe this question is addressed in the experiments; there is no baseline which uses to the same X (the Book11K corpus). Thus, it is not clear if the reported results are simply as a result of (a) using a lot more data, or (b) the model is capturing something interesting beyond superficial word co-occurence statistics.I would have liked to see a comparison to a strong baseline which uses the same training data. Short of this, it would have at least been good to see a *weak* baseline, e.g. a bag-of-words model. Yet another option would have been to train word embeddings on Book11K, and use these in some of the more engineered baselines so that at least the data conditions are matched.One common issue with RNNs in the encoder-decoder framework is the encoder must summarize the entire input in a single vector. One heuristic that has been used to alleviate this problem is to reverse the order of the tokens. More recently, attention models have been used. I would have liked to see a little discussion of this issue. In a similar vein, I would have liked to see more details about:-how the proposed approach was tuned, and how sensitive it is to hyperparameters-what alternate architectures were used, if any. For instance, an obvious alternative is to only predict the next sentence given the current sentence (it's not 100% obvious why it makes sense to predict sentences in reverse, even though I'm willing to believe it's helpful as it doubles the amount of data).-how performance improves with increasing amounts of unlabeled data X (presumably could be derived from model checkpoints from a prior training run).UPDATE AFTER AUTHOR RESPONSE:Including a baseline that uses word-level embeddings trained on Book11K would improve the paper. However, it seems to me that the natural baseline to compare to is the paragraph vector model [1], with matched train and test conditions. I don't understand why this comparison wasn't made; perhaps this should be clarified in the paper.[1] http://arxiv.org/pdf/1405.4053v2.pdf


Analysis of Robust PCA via Local Incoherence
We investigate the robust PCA problem of decomposing an observed matrix into the sum of a low-rank and a sparse error matrices via convex programming Principal Component Pursuit (PCP). In contrast to previous studies that assume the support of the error matrix is generated by uniform Bernoulli sampling, we allow non-uniform sampling, i.e., entries of the low-rank matrix are corrupted by errors with unequal probabilities. We characterize conditions on error corruption of each individual entry based on the local incoherence of the low-rank matrix, under which correct matrix decomposition by PCP is guaranteed. Such a refined analysis of robust PCA captures how robust each entry of the low rank matrix combats error corruption. In order to deal with non-uniform error corruption, our technical proof introduces a new weighted norm and develops/exploits the concentration properties that such a norm satisfies.

Summary of paper:Until recently, the related but different problems of matrix completion and robust PCA - via nuclear norm minimization - had similar assumptions under which recovery was known: incoherence of the low-rank matrix, and uniform randomness in the samples (for completion) or errors (for robust PCA). Recently however it was shown that for matrix completion recovery is provably possible, via same convex methods, even when the sampling is non-uniform, provided this non-uniformity is adapted to LOCAL coherence. This paper establishes a similar result for robust PCA, showing that if the errors are non-uniromly random (again in a way that is adapted to the same local coherences) then exact recovery is again possible.Quality:Overall the result is likely to be important and of interest. The techniques are new, but along lines somewhat similar to those employed in the (recent) matrix completion analysis using local coherence.The experiments are not very convincing because the low-rank matrices are constructed to have very similar local coherences. It would have been better to see matrices with very different local coherences. This can be done by pre and post multiplying the low-rank matrices generated in the paper with diagonal matrices that have large dynamic range on the diagonal elements. Clarity:The paper is clearly written, though by the very nature of what it is trying to do is a bit dense to read.Originality:The paper takes the new recent understanding of how local coherence can allow for non-uniformity in sampling for matrix completion, and extends it to the case of errors in robust PCA. The results are original.Significance:Robust PCA has a huge number of applications, and by this token an improvement in its understanding is significant. However the paper does not provide any new method; but rather a better analysis of the already popular method. So its empirical significance may be lower.It would have been nice to see specific examples where this kind of non-uniformity both naturally arises, and leads to appreciably better recovery. For example, does this imply that for the graph clustering problem with unequal size clusters, larger clusters can be sparser but still recover-able ?


Extending Gossip Algorithms to Distributed Estimation of U -Statistics
Efficient and robust algorithms for decentralized estimation in networks are essential to many distributed systems. Whereas distributed estimation of sample mean statistics has been the subject of a good deal of attention, computation of Ustatistics, relying on more expensive averaging over pairs of observations, is a less investigated area. Yet, such data functionals are essential to describe global properties of a statistical population, with important examples including Area Under the Curve, empirical variance, Gini mean difference and within-cluster point scatter. This paper proposes new synchronous and asynchronous randomized gossip algorithms which simultaneously propagate data across the network and maintain local estimates of the U -statistic of interest. We establish convergence rate bounds of O(1/t) and O(log t/t) for the synchronous and asynchronous cases respectively, where t is the number of iterations, with explicit data and network dependent terms. Beyond favorable comparisons in terms of rate analysis, numerical experiments provide empirical evidence the proposed algorithms surpasses the previously introduced approach.

This paper is about computing higher-order statistics measured over pairs of observations, where the observations themselves are distributed across a network of nodes. These U-statistics include many quantities of interest such as the sample variance, correlation and scatter coefficients, and other aggregate measures. The main goal of the paper is to propose a gossip-style decentralized aggregation/averaging algorithm to compute these statistics.Overall, this is a nice contribution to the literature on gossip algorithms, but the authors should make a better connection to existing approaches and ideas.This is a "heavy" review. I did not have time to verify the correctness of the material in the supplement.Comments1) The main trick on the algorithm side is that when two nodes exchange their estimate values and average them, they also swap a secondary state variable representing their initial observation. This means that the initial observations are each performing a random walk (albeit coupled). Mentioning this more clearly in the main manuscript would help and also would motivate the analysis method via the multiple copies of the graph.2) There has been a lot of work on gossip approaches for computing statistics beyond the references cited by the authors. A snapshot of the state of the art from a few years ago is the Dimakis et al. Proc. IEEE paper. Situating the current work in the context of the works mentioned there might be more helpful than the somewhat cursory explanation in the current manuscript.3) On page 4 it might be worth clarifying what the induced distribution on edges is from the "node wakes up, contacts neighbor" process. The phrase "drawing a random edge of the network" does not specify the distribution.4) There is a natural other approach to this problem, which is to allow more memory at each node and essentially use the same contact process to have each node flood its own value across the network. Nodes store the received observations (tagged with the node who initially possessed it). At each time nodes can calculate the U statistic on the subset of the data that they possess. How would this compare to the current method? This may help show why the averaging can be helpful, or show the loss in performance when the memory at each node is limited. Without this comparison it's not clear that the rate 1/t is good or bad. For expanders the flooding method may be quite good.5) The authors should take a look at the work on lifting constructions for accelerating gossip algorithms, as it seems quite related to their construction in the paper. See for example:K. Jung, D. Shah, and J. Shin, "Fast gossip through lifted Markov chains," in Proc. Allerton Conf. on Comm., Control, and Comp., Urbana-Champaign, IL, Sep. 2007. W. Li and H. Dai, "Location-aided fast distributed consensus," in IEEE Transactions on Information Theory 56(12), 2010. F. Chen, L. Lovasz, and I. Pak, "Lifting markov chains to speed up mixing," in Proceedings of the thirty-first annual ACM symposium on Theory of computing. ACM, 1999, pp. 275-281.6) Analysis of gossip algorithms tend to make a worst-case assumption on the initial data distribution. That is, the guarantee is over all initial data vectors $X$. This is not clear in the problem statement of the statement of the theorem. What is the assumption on the data distribution and what are the guarantees on the error as a function of the initial data?Small things - p1: The bipartite definition using (S \times T) \cup (T \times S) implies directed graphs. - p4: "let us stress out that" --> "we stress that" - p4: "awakes and contact" -- > "awakes and contacts" - p6: "requires to initiate" --> "requires initiating" - p7: "allows to create" --> "allows us to create"AFTER AUTHOR FEEDBACK: I think the clarifications and modifications suggested by the authors are good. I am revising my initial score of 7 up to an 8.I think that they should still take a close look at the lifting constructions (esp. LADA) and modify the related work.Furthermore, the experiments should include the baseline comparisons.


Logarithmic Time Online Multiclass prediction
We study the problem of multiclass classification with an extremely large number of classes (k), with the goal of obtaining train and test time complexity logarithmic in the number of classes. We develop top-down tree construction approaches for constructing logarithmic depth trees. On the theoretical front, we formulate a new objective function, which is optimized at each node of the tree and creates dynamic partitions of the data which are both pure (in terms of class labels) and balanced. We demonstrate that under favorable conditions, we can construct logarithmic depth trees that have leaves with low label entropy. However, the objective function at the nodes is challenging to optimize computationally. We address the empirical problem with a new online decision tree construction procedure. Experiments demonstrate that this online algorithm quickly achieves improvement in test error compared to more common logarithmic training time approaches, which makes it a plausible method in computationally constrained large-k applications.

Heavy reviewSummary: The paper concerns multi-class problems with a large number of classes. It introduces a novel label tree classifier that learns and predicts in logarithmic time in the number of classes. Theoretical guarantees in terms of a boosting-like theorem have been proven. Moreover, not only node classifiers, but also the structure of the tree is trained online. Additionally, the authors show a simple subtree swapping procedure that ensures proper balancing of the tree. Empirical results show attractiveness of the introduced approach.Quality:The paper presents both theoretical and empirical results. The method is clearly derived and theoretically justified. Some minor comments are given below.The objective after reformulation can be stated as:J(h) = 2 \sum_{i=1}^k | \Pr(i) \Pr(h(x) > 0) - \Pr(i, h(x) > 0) |This is a kind of an independence test between i and h(x) > 0. I suppose that such measures have been used in rule induction (e.g., in generation of association rules) and decision tree learning. This is also similar to the so-called Carnap's confirmation measure (see, "Comparison of confirmation measures", Cognition, 2007, and "Logical foundations of probability (2nd ed.).", 1962).Decision tree learning can be introduced as a recurrent procedure of empirical risk minimization guided by a surrogate loss function, where both splits and leaf predictions are computed as empirical risk minimizers (the notion of a purity measure is not needed then). For C4.5, this is logistic loss, and for CART squared error loss. It would be interesting to check whether there exists a loss function that corresponds to J(h).The authors do not clearly describe the difference between LOMTree and standard decision trees. Footnote 4 states that standard criteria such as Shannon or Gini entropy satisfy requirement of balanced and pure splits. Since it is possible to adapt standard decision trees to the problems with a large number of classes [16,17], it is not clear what is the difference between these two approaches. Can we substitute J(h) by Shannon entropy?I suppose that there exists a kind of a trade-off between test-time complexity and the complexity of the function class, i.e., by minimizing the number of prediction steps we should probably extend the complexity of the function class. This would explain the empirical results of OAA and LOMTree. Could you comment on this?The prediction time results are given per test example. The overall method is also designed for improving test time for a single test example. However, when test examples are coming in mini-batches different tricks can be used to improve test time. How to use this method in such scenario? Clarity:The paper is clearly written. Originality:The paper is very original as it clearly states the problem of efficient learning and prediction in case of large output spaces. The introduced algorithm is also novel, however, the authors should take into account that the problem of decision tree induction was exhaustively studied in 80' and 90'. Many algorithms have already been introduced that share similarities with LOMtrees (incremental learning of trees, trees for data streams, Hoeffding trees, trees with linear splits). Significance:The paper will be (already is) very influential for large-scale machine learning.After rebuttal:Thank you for your responses. I gave the references to confirmation measures since this was the most similar measure I knew to your objective. It would be great if you will explore this topic in the extended (e.g., arxiv or journal) version of the article. I have found your objective very interesting, but I think that similar measures have already been considered in many settings and many interesting links can be found. I agree that such measures are not so popular and successful in decision tree learning, especially in the multi-class setting.The same concerns incremental learning of decision trees. There are some approaches that you can easily find in Internet. The question is how they relate to your approach. One well-know approach are Very Fast Decision Trees (aka Hoeffding trees) introduced by Domingos and Hulten (2000). A detailed discussion can be postponed however to the extended version of your paper.


Precision-Recall-Gain Curves: PR Analysis Done Right
Precision-Recall analysis abounds in applications of binary classification where true negatives do not add value and hence should not affect assessment of the classifier's performance. Perhaps inspired by the many advantages of receiver operating characteristic (ROC) curves and the area under such curves for accuracybased performance assessment, many researchers have taken to report PrecisionRecall (PR) curves and associated areas as performance metric. We demonstrate in this paper that this practice is fraught with difficulties, mainly because of incoherent scale assumptions -e.g., the area under a PR curve takes the arithmetic mean of precision values whereas the F β score applies the harmonic mean. We show how to fix this by plotting PR curves in a different coordinate system, and demonstrate that the new Precision-Recall-Gain curves inherit all key advantages of ROC curves. In particular, the area under Precision-Recall-Gain curves conveys an expected F 1 score on a harmonic scale, and the convex hull of a PrecisionRecall-Gain curve allows us to calibrate the classifier's scores so as to determine, for each operating point on the convex hull, the interval of β values for which the point optimises F β . We demonstrate experimentally that the area under traditional PR curves can easily favour models with lower expected F 1 score than others, and so the use of Precision-Recall-Gain curves will result in better model selection.

These authors demonstrate that traditional Precision-Recall (PR) curves and associated area under PR curve are fraught with difficulties.They show how to fix this by plotting PR curves in a different coordinate system, and propose Precision-Recall-Gain curves which inherit all key advantages of ROC curves. This study is of interest to the readership and suitable for the NIPS audience. This innovative study will advance the field of performance assessment for predictive models and diagnostic tests. However, several problems are noted in the methods and clarity of presentation. a. At line 390, what are AUROCs for IBK and Logistic Regression? b. It is difficult to interpret the information embedded in Figure 5.I would recommend using a table to report the outcomes for five Weka models trained on 19 UCI datasets, and using Figure 5 to display performance for several examples including the mushroom dataset.c. For Figure 5, it is better to avoid abbreviations and to provide detailed explanation so that it can stand alone. It is also better to provide the definition for IBK since it isn't a widely used model.


Newton-Stein Method: A Second Order Method for GLMs via Stein's Lemma
We consider the problem of efficiently computing the maximum likelihood estimator in Generalized Linear Models (GLMs) when the number of observations is much larger than the number of coefficients (n p 1). In this regime, optimization algorithms can immensely benefit from approximate second order information. We propose an alternative way of constructing the curvature information by formulating it as an estimation problem and applying a Stein-type lemma, which allows further improvements through sub-sampling and eigenvalue thresholding. Our algorithm enjoys fast convergence rates, resembling that of second order methods, with modest per-iteration cost. We provide its convergence analysis for the case where the rows of the design matrix are i.i.d. samples with bounded support. We show that the convergence has two phases, a quadratic phase followed by a linear phase. Finally, we empirically demonstrate that our algorithm achieves the highest performance compared to various algorithms on several datasets.

The authors uses an estimation approach to estimate the scaling matrix and replaces an O(np^+p^3) per iteration cost with an O(np+p^2) cost.The new optimization method applies to generalized linear models and is extremely well crafted.  The paper is extremely clear, the quality is superb, the originality is good and the significance only limited by the experiments and the importance of generalized linear models.The experiments are a bit weak.There are only two examples -- logistic regression and ordinary least squares.These are compared to general optimization techniques, but not to any methods tailored for these.There was for example a paper by Byrd, Oztoprak and Nocedal that used sampling estimates of the Hessian for logistic regression.This is quite closely related to what's done in this paper.There are also other papers using sampling techniques to estimate the Hessian.It is a bit disconcerting that the "appendix" is 3 times longer than the paper itself.In many ways the paper is just an advertising pamphlet for the appendix, but the main body of the paper is well crafted and most readers may not wish to dive into the most horrid derivations in the appendix.Here are some detailed comments: "Distribution of a random variable belongs to an..." --> "The distribution of a random variable belongs to an..."I expected to see the dependence of t shown on the RHS in the first equation in chapter 3.As it stands it's just an unnecessary repetition of 2.3."well-approximates": this is not a word.The "event" E defined in 4.1 is not properly defined.I imagine this would be a set over \beta^{t} in which the set should be written\{\beta^{t}: |\mu_2(\beta^{t})+...| > \tau,...\} The way it is written, it could also be a set over tau, beta_*, Sigma etc.  What's the difference between ||\beta^t-\beta_*|| and E[||\beta^t-\beta_*||] in Lemma 4.1, and what is the expectation over? Given that the proof is in the lengthy appendix, an explanation/discussion would be helpful.Could you define sub-gaussian in the main body of the paper?Figure 2 is a bit busy and I can't tell which line is which as some of the colors are very close and the legend is not sufficiently good to match the line up with the legend.The bibliography has not been done in the usual NIPS style.I think the authors need to conform with the usual standard NIPS template.


Beyond Sub-Gaussian Measurements: High-Dimensional Structured Estimation with Sub-Exponential Designs
We consider the problem of high-dimensional structured estimation with normregularized estimators, such as Lasso, when the design matrix and noise are drawn from sub-exponential distributions. Existing results only consider sub-Gaussian designs and noise, and both the sample complexity and non-asymptotic estimation error have been shown to depend on the Gaussian width of suitable sets. In contrast, for the sub-exponential setting, we show that the sample complexity and the estimation error will depend on the exponential width of the corresponding sets, and the analysis holds for any norm. Further, using generic chaining, we show that the exponential width for any set will be at most √ log p times the Gaussian width of the set, yielding Gaussian width based results even for the sub-exponential case. Further, for certain popular estimators, viz Lasso and Group Lasso, using a VCdimension based analysis, we show that the sample complexity will in fact be the same order as Gaussian designs. Our general analysis and results are the first in the sub-exponential setting, and are readily applicable to special sub-exponential families such as log-concave and extreme-value distributions.

The author considers the regression problem, where one is given$$ y = X \theta^* + \omega $$for some design matrix $X$ and noise vector $\omega$. The goal is to recover some estimate $\hat{\theta}$ of $\theta^*$ such that $\|\hat{theta} - \theta^*\|_2$ is small with high probability (over the noise vector $\omega$ as well as randomness in the design matrix $X$). The author considers the standard approach of recovering $\hat{\theta}$ as the solution to an optimization problem (a norm-regularized least squares regression).Much attention has been given to this problem in the past when $X$ and $\omega$ have subgaussian entries. In that case, the number of rows $n$ of the design matrix $X$ that is needed relates to the gaussian mean width of the "error set" $A$ defined in lines 108-113 of the paper. In this submission, attention is given to the case when "subgaussian" is replaced with "subexponential". In this case, $n$ relates to the exponential width of $A$. A theorem is given (Theorem 1) stating that exponential width is at most gaussian width times $\sqrt{\log p}$ for any subset T of $\mathbb{R}^p$. This theorem is a simple corollary of two theorems from Talagrand's book after computing the exponential width of the unit $\ell_1$ ball, which is very standard (Theorems 1.2.7 and 2.6.2 in that book; Theorem 5.2.7 is also cited in the submission, but it is irrelevant, it provides the lower bound on the exponential width in terms of the $\gamma$-functionals, but that lower bound is never used in any of the proofs in the submission, and furthermore the lower bound is only true anyway when the r.v.'s *are* exponential and not merely subexponential, unlike this submission).Technically, I found the jump from analyzing the gaussian case in previous work to the subexponential case in this submission to be quite a small jump. The new main theorem, Theorem 1, follows essentially immediately from the two theorems cited in Talagrand's book.In any case, not having a big technical contribution is perfectly OK. More importantly, I think the submission could benefit greatly from more time spent on motivation in the introduction. My perhaps wrong impression is that one can sometimes choose their design matrix, and in those cases they can choose their entries to be subgaussian to get away with few measurements (depending on the gaussian width of the error set). In these cases where a choice is present, it seems there is no motivation to choose to use a design matrix with subexponential entries since the width parameter only gets worse. The question then is: are there strong motivating examples when you cannot choose your design matrix (it is simply given to you by the real world) *AND* it makes sense to model the entries in that design matrix as being subexponential? As far as I can see, the main motivation for this submission hinges upon a positive answer to this question, yet this question does not seem to be addressed adequately in the introduction. (The answer may very well be yes, but I have no idea, and the author should spend considerable time on concrete such examples in the introduction if they exist, since it justifies the importance of the whole paper.)OTHER COMMENTS:FROM THE MAIN PAPER: * line 070: "Note that sub-exponentials are the class of distributions which have heavier tails compared to sub-Gaussians and for which all moments exist.". I don't agree with the word "the" here before "class", since it is one of many. The distribution with pdf e^{-|x|^{1.5}} has heavier tails than sub-gaussians with all moments existing also.* line 071: "Distributions having heavier tails than sub-exponentials start losing moments." This doesn't sound right to me. What about a distribution with pdf e^{-sqrt(|x|)}? The integral of x^p e^{-sqrt(x)} from 0 to infinity converges for any constant p greater than 0 (i.e. all moments exist). * Lines 084 and 085, $A$ is used without yet being defined and is thus somewhat confusing. It is only clear later in line 109 what $A$ was referring to. * I'm confused by line 115 which presents an inequality for $\lambda_n$. $\lambda_n$ is something you set in (2) so that the solution to the optimization problem has some desired property (e.g. it's close to $\theta^*$ in some norm with high probability). Are you saying it should be set to satisfy the given inequality? And are you saying there's a *specific* $\beta$ bigger than 1 for which it should be set this way? I cannot understand what this line means as written (although I guessed the earlier part of this bullet based on line 108). * Lines 199-201, it states (4) holds for any g whose entries have subgaussian decay. This is false. The upper bound of (4) is indeed true as long as there is subgaussian decay, but the lower bound only holds for gaussian decay, not subgaussian. e.g. Rademachers are subgaussian, but gamma_2 is not a lower bound for Rademachers (example: gamma_2 of the ell_1 ball in p dimensions is $\Theta(\sqrt{\log p})$, but the Rademacher width equals $1$). * Lines 210-212 claim that (5) holds for any subexponential r.v.'s. This is again false for the same reason as the last bullet (also see my last comment below from the supplementary file).FROM THE SUPPLEMENTARY FILE: * In line 124, "lose" should be "loose". Also, "For e.g." is redundant; it should just be "e.g." without the "for". * Equation (16) is slightly confusing. The "K" here is an absolute constant and should not be confused with the K from Lemma 1 (it seems the K from Lemma 1 is assumed to equal 1 here). * Below line 215, it states "the result is applicable to any process that has concentration inequality (16)". This is false. The upper bound of (17) is indeed true as long as (16) holds, but the lower bound is only true for the case when the entries of X are exactly exponential (as opposed to subexponential). For example, gaussians are subexponential, but the lower bound is wrong if the entries of X are actually gaussians (the gamma_1 term shouldn't be there). For a more trivial example, the distribution that is supported only on the number "zero" is subexponential, and the lower bound is obviously false for it (although I guess this trivial example doesn't quite work if you assume all variances are 1).