Adaptive Newton Method for Empirical Risk Minimization to Statistical Accuracy
We consider empirical risk minimization for large-scale datasets. We introduce Ada Newton as an adaptive algorithm that uses Newton's method with adaptive sample sizes. The main idea of Ada Newton is to increase the size of the training set by a factor larger than one in a way that the minimization variable for the current training set is in the local neighborhood of the optimal argument of the next training set. This allows to exploit the quadratic convergence property of Newton's method and reach the statistical accuracy of each training set with only one iteration of Newton's method. We show theoretically that we can iteratively increase the sample size while applying single Newton iterations without line search and staying within the statistical accuracy of the regularized empirical risk. In particular, we can double the size of the training set in each iteration when the number of samples is sufficiently large. Numerical experiments on various datasets confirm the possibility of increasing the sample size by factor 2 at each iteration which implies that Ada Newton achieves the statistical accuracy of the full training set with about two passes over the dataset.
(i) Global convergence of Newton's method requires implementation of a line search subroutine and line searches in ERM require a complete pass over the dataset. (ii) The quadratic convergence advantage of Newton's method manifests close to the optimal solution but there is no point in solving ERM problems beyond their statistical accuracy. (iii) Newton's method works for strongly convex functions but loss functions are not strongly convex for many ERM problems of practical importance. (iv) Newton's method requires inversion of Hessians which is costly in large dimensional ERM.
Because stochastic Newton-like methods can't use line searches [cf. (i)], must work on problems that may be not strongly convex [cf. (iii)], and never operate very close to the optimal solution [cf (ii)], they never experience quadratic convergence. They do improve convergence constants and, if efforts are taken to mitigate the cost of inverting Hessians [cf. (iv)] as in [21,12,7,18] they result in faster convergence. But since they still converge at linear rates they do not enjoy the foremost benefits of Newton's method.
In this paper we attempt to circumvent (i)-(iv) with the Ada Newton algorithm that combines the use of Newton iterations with adaptive sample sizes [5]. Say the total number of available samples is N , consider subsets of n  N samples, and suppose the statistical accuracy of the ERM associated with n samples is V n (Section 2). In Ada Newton we add a quadratic regularization term of order V n to the empirical risk -so that the regularized risk also has statistical accuracy V n -and assume that for a certain initial sample size m 0 , the problem has been solved to its statistical accuracy V m0 . The sample size is then increased by a factor ↵ > 1 to n = ↵m 0 . We proceed to perform a single Newton iteration with unit stepsize and prove that the result of this update solves this extended ERM problem to its statistical accuracy (Section 3). This permits a second increase of the sample size by a factor ↵ and a second Newton iteration that is likewise guaranteed to solve the problem to its statistical accuracy. Overall, this permits minimizing the empirical risk in ↵/(↵ 1) passes over the dataset and inverting log ↵ N Hessians. Our theoretical results provide a characterization of the values of ↵ that are admissible with respect to different problem parameters (Theorem 1). In particular, we show that asymptotically on the number of samples n and with proper parameter selection we can set ↵ = 2 (Proposition 2). In such case we can optimize to within statistical accuracy in about 2 passes over the dataset and after inversion of about 3.32 log 10 N Hessians. Our numerical experiments verify that ↵ = 2 is a valid factor for increasing the size of the training set at each iteration while performing a single Newton iteration for each value of the sample size.

The authors study the ERM (Empirical Risk Minimization) problem and propose an Adaptive Newton algorithm for solving it to statistical accuracy. The idea is to consider a sequence of nested ERM problems of increasing sizes (each new ERM problem is described by all the examples of the previous ERM problem in the sequence and a number of fresh samples) up to a certain fixed size, and take a carefully designed Newton step (of unit stepsize) related to each ERM problem. The claim is that if this is done “right”, each ERM problem is solved up to statistical accuracy.
The paper contains a novel and interesting idea, and is well written. I think it should be accepted.    This is one of a very few papers which attempt to combine optimization and statistical considerations. Most papers suggesting algorithms for solving the ERM problem simply focus on solving the deterministic ERM problem, and ignore the fact that ERM objective is an approximation to the true loss which arises as an expectation of the loss over an unknown sample distribution; and hence is subject to an approximation error. This is of course known in the literature, but it is notoriously difficult to address both the optimization aspect and the approximation aspect in a meaningful way in a single work. This papers takes a highly novel route to this problem; and I think there is a potential in this line of work.   However, the paper is not flawless – below I will enumerate some of the issues with the paper. Some of these issues can be remedied by a revision, but some seem to be an intrinsic part of the proposed approach, and hard to get rid off.  1) The introduction enumerates 4 challenges left in the implementation of Newton-like methods for ERM. I agree that these issues remain with most of the approaches. However, there are approaches which circumvent these issues – but these are not mentioned by the authors. The paper would be stronger if a more balanced treatment was made. For instance, consider the SDNA algorithm of Qu, Richtarik, Takac and Fercoq (Proceedings of The 33rd International Conference on Machine Learning, pp. 1823–1832, 2016). For L2-reglarized ERM problems, this method takes Newton-like steps in random subspaces in the dual. i) The method has a global linear rate, better than SDCA, without any explicit line-search procedure.  ii) Since there is no point in looking at local convergence beyond statistical accuracy, the analysis of SDNA focuses on speedup from mini-batching, which is more important. This speedup is superlinear in the  mini-batch size, unlike all other stochastic first-order methods. iii) SDNA was analysed for strongly convex ERM problems – so this issue is present. iv) This issue is circumvented by inverting sufficiently small random principal submatrices of the Hessian in the dual (the Hessian does not change in the dual, which is the advantage of using the dual formulation). This is just one example of how *some* of these issues *are* addressed by existing research. Hence, the paper should be more precise in stating the current status of these challenges.  2) I do not see how the claim “Our theoretical and numerical analyses indicate that we
can make alpha = 2. In that case, we can can optimize to within statistical accuracy in about 2 passes over
the dataset and after inversion of about 3.32 log10 N Hessians. “ is supported in theory.
 Also, it is not clear enough what exactly the above statement means. It is made in the abstract, on lines 49-51 and lines 143-145. The precise complexity depends on values values of c and alpha, which in turn depend on the values of V_n, which are unknown in practice. The sizes of the constants in the O(1/n), O(1/sqrt{n}) decay rates will matter, and these will influence the precise constant elsewhere, including the constant 2 claimed above. So, I think this conclusion is exaggerated and needs moderation.  3) The same applies to the claim that the result is supported by experiments. Conclusions can’t be drawn from a single experiment using a single dataset (The sentence “We have observed similar performances for other datasets such as A9A,
COVTYPE, and SUSY.
” is much less satisfying than including the experiments). Another reason to moderate the statement and remove it from the abstract in order for the readers not to be potentially misled. 4) The authors should make a detailed complexity analysis of the method, including the cost of each iteration, and compare this to existing results. A the moment, no such comparison is given, and hence it is not straightforward how the results should be interpreted and why they should be appreciated. Are your complexity results better than those of existing methods, such as SAGA or SDCA or Quartz? 5) The algorithm seems to be conceptual in practice, and impractical (this is admitted on lines 126-127). This is because the method relies on the knowledge of parameters such as V_n for all n, but these are not known. Fine tuning these might be computationally difficult, and I would expect the actual solver to be slower than state of the art (SDCA, SAGA, Quartz, SAG, AdaSDCA, dfSDCA, SVRG, S2GD, Finito, …)  on most datasets for this reason. There are more parameters which need to be tuned, such as alpha_0, beta, c. How would these be chosen in practice? Will the method be robust to this choice of parameters across different datasets? How will it compare to off-the shelf implementation of SDCA or S2GD, say, which do not require any parameters to be tuned? If the authors believe the method to be practical, which would enhance the paper considerably, then considerations of this type need to be included in the paper, and extensive computational experiments performed.  6) Theorem 1 (appearing in Section 3) relies on assumptions which are not stated until Section 4. The reader is not alerted to this. Likewise, Proposition 2 relies on these assumptions, but this is not explicitly stated. Moreover, the term M appears in the result there, unexpectedly, as this is the first time this term appears in the paper. I later found tat it was defined in a later section. Some adjustment is needed to the text in order to rectify the flow of ideas a bit and make the paper a bit easier to read in these places (the paper is very well written already; this is a minor issue which can be easily addressed in a number of ways).    Small issues:  7) Line 50: cases -> case 8) Should not c be a parameter of Alg 1? 9) In several paces (e.g., lines 171, 189, …), you write “Assumption 1-3” instead of “Assumptions 1-3” 10) Line 178: that -> in which (might be best to break this sentence up into 2-3 shorter sentences) 11) Line 181: necessary -> necessarily 12) Line 264: The -> the    Bibliography: There are many issues with the bibliography. Here are some I found:  1) [1]: capitalize journal name 2) [3]: capitalize Newton 3) [4]: include NIPS volume 4) [6]: appeared already in Proceedings of The 33rd International Conference on Machine Learning, pp. 1463–1471, 2016 5) [7]: Saga -> SAGA; also include NIPS volume 6) [10]: Bfgs -> BFGS; appeared: Proceedings of The 33rd International Conference on Machine Learning, 1869-1878, 2016  7) [11]: newton -> Newton 8) [12]: include NIPS volume 9) [13]: l-bfgs -> L-BFGS; appeared: Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, pp. 249–258, 2016 10) [14] publisher?; volume I, not volume i 11) [15]: was published already (long time ago) 12) [18]: include NIPS volume 13) [19]: newton -> Newton 14) [23]: capitalize book title  --- post rebuttal comments ---  Thanks for the explanations; some of my concerns were addressed, but many were not (or not in the way I hoped). I still think highly of the paper (and think it should be accepted), but to be fair, I also believe the score I gave for clarity is higher than it really should be. I am decreasing the clarity score from 4 to 3.


Verification Based Solution for Structured MAB Problems
We consider the problem of finding the best arm in a stochastic Multi-armed Bandit (MAB) game and propose a general framework based on verification that applies to multiple well-motivated generalizations of the classic MAB problem. In these generalizations, additional structure is known in advance, causing the task of verifying the optimality of a candidate to be easier than discovering the best arm. Our results are focused on the scenario where the failure probability must be very low; we essentially show that in this high confidence regime, identifying the best arm is as easy as the task of verification. We demonstrate the effectiveness of our framework by applying it, and matching or improving the state-of-the art results in the problems of: Linear bandits, Dueling bandits with the Condorcet assumption, Copeland dueling bandits, Unimodal bandits and Graphical bandits.

This paper proposes a verification-based framework for solving a range of bandit problems, including condorcet dueling bandits, copeland dueling bandits, linear bandits, unimodal bandits, and graphical bandits.  The setting considered is PAC-style guarantees for pure exploration, rather than online regret minimization.  The authors show an iterative approach whereby a verification algorithm is used to verify a proposed solution by an exploration algorithm.  Since the verification algorithm has cheaper sample complexity, one can construct a meta-algorithm that iteratively calls the two algorithms in order to achieve rigorous PAC-style sample complexity guarantees.  These guarantees are compared against existing results, and the authors demonstrate improvements in some regimes.  In general, I think the technical results are interesting and mostly sound, although I can't help but feel that the presentation could be significantly improved.  Some of the results seem a bit incomplete, although I think a more complete theoretical account is straightforward.    My larger concern is the significance of the results in the paper.  Part of this could be resolved by greater clarity in the writing.  More details are covered in the sections below.  As it stands, my review is borderline.  I can see this paper as suitable for NIPS or a bit below the accept threshold, depending on the author response.
*** Significance Part 1 *** I wonder if the authors have looked into the details of the proofs of some of the previous work.  One common trick that people do in these types of theoretical analyses is to simplify the results because one cares only about the simpler bounds that merges together a more complicated summation of terms into a single term.  For example, the result for Linear Bandits shows an improvement that could be non-existent with a more refined analysis of the result in [19].  This would be somewhat analogous to what the authors noted in footnote 3.  *** Significance Part 2 *** The authors argue that these results are meaningful improvements upon previous work.  However, it would be easier to accept this argument if the authors can construct explicit cases that satisfy the results.  For instance, I'm particularly interested in the linear case.   Numerical experiments could also help and/or complement the theoretical results.  *** Clarity *** The paper feels a bit incomplete in its writing.  For instance, Copeland dueling bandits isn't substantially discussed at all in the main paper.  And related to the above point on significance, the reader feels a bit disappointed by a lack of substantial discussion on the implications of the theoretical results.  Furthermore, the paper doesn't actually formally define any problem settings.  I realize this is very difficult due to the generality of the framework being proposed, but this lack of specificity is somehow dissatisfying.  *** Proofs *** I'm curious about the equations after Line 135. It seems like there is an extra kappa*H_explore term -- can the authors explain this?  Perhaps related to the above question, the authors never explicitly state a sample complexity of the verification oracle when the findbestarm subroutine fails.  For instance, Lemma 14 doesn't have a sample complexity guarantee.  I think this should be a straightforward fix, but the theoretical results are incomplete without them.


Path-Normalized Optimization of Recurrent Neural Networks with ReLU Activations
We investigate the parameter-space geometry of recurrent neural networks (RNNs), and develop an adaptation of path-SGD optimization method, attuned to this geometry, that can learn plain RNNs with ReLU activations. On several datasets that require capturing long-term dependency structure, we show that path-SGD can significantly improve trainability of ReLU RNNs compared to RNNs trained with SGD, even with various recently suggested initialization schemes.

In this work the authors present an adaptation of the Path-SGD algorithm for RNNs. They characterize all node-wise invariances in RNNs and show that the update rule for their modified algorithm is invariant to all feasible node-wise rescalings. They provide experiments which indicate that their modified optimization method yields improvements for plain RNNs trained with existing methods, although they still lag behind LSTMs.
The authors frame their work as a step towards bridging the gap between plain RNNs and more complex LSTMs/GRUs by using an optimization method which accounts for the geometry of the loss surface. This seems to be a worthwhile goal (since plain RNNs are computationally cheaper and easier to analyze theoretically) and their experiments show some promising results in improving performance over plain RNNs trained with existing optimization methods.   However, it is not clear to me how the method that the authors use in practice differs significantly from regular Path-SGD introduced in previous work. The authors do present an adaptation of Path-SGD to networks with shared weights, and show that the new rescaling term applied to the gradients can be divided into two terms k1 and k2. But then, they note that the second term, which accounts for interactions between shared weights along the same path, is expensive to calculate for RNNs and show some empirical evidence that including it does not help performance. In the rest of the experiments, they ignore the second term, which to my understanding is essentially what makes the method introduced here different from regular Path-SGD. Isn't ignoring the second term the same as simply applying Path-SGD as if the weights were not shared, and combining the rescaling terms corresponding to shared weights? Also, part of the justification for this work in Section 3 is that RNNs with ReLUs can encounter the "exploding activation" problem due to the fact that ReLUs do not saturate, and that taking into account the interactions between shared weights along the same path can avoid this problem. I would have thought the second term would be necessary to avoid this. I would like the authors to clarify the major differences, if any, between the approximation that they use in the experiments (which ignores the second term) and the Path-SGD method introduced in other work, and its relation to the exploding activation problem in ReLU-RNNs. I would be willing to raise my score if the authors can explain how the method that they use in practice is significantly different than the previous Path-SGD algorithm.  The experiments do indicate that using path normalization offers a consistent improvement for plain RNNs on language modeling and sequential MNIST, which is encouraging, although they have yet to catch up to the LSTM.   A few remarks: -Section 5.1: it would be better to report experiments on the same tasks that are used in evaluation, i.e. character level rather than word-level experiments.   - for addition problem experiments, it seems like the authors initialize the RNN with identity transition matrix (this could be made more clear). Note that this is close to the exact solution presented in "Recurrent Orthogonal Networks and Long Memory Tasks", Henaff et al. 2016, and gives the RNN an advantage. Also, the experiments in that paper show that LSTMs can solve the addition task for T=750. For fair comparison with LSTMs, the RNN should be initialized randomly. -Line 217: both version -> both versions


Solving Marginal MAP Problems with NP Oracles and Parity Constraints
Arising from many applications at the intersection of decision-making and machine learning, Marginal Maximum A Posteriori (Marginal MAP) problems unify the two main classes of inference, namely maximization (optimization) and marginal inference (counting), and are believed to have higher complexity than both of them. We propose XOR_MMAP, a novel approach to solve the Marginal MAP problem, which represents the intractable counting subproblem with queries to NP oracles, subject to additional parity constraints. XOR_MMAP provides a constant factor approximation to the Marginal MAP problem, by encoding it as a single optimization in a polynomial size of the original problem. We evaluate our approach in several machine learning and decision-making applications, and show that our approach outperforms several state-of-the-art Marginal MAP solvers.

This paper aims at approximating marginal MAP (MMAP) problem instances using NP oracles. In the spirit of Ermon et. al.’s work (ICML 2013) for approximating #P-complete problems using pairwise independent hashing and optimization, the authors propose a randomized algorithm (XOR-MMAP) that transforms an MMAP query into a series of joint MAP (or MPE) queries, each begin defined as a nonlinear optimization problem over all variables, subject to randomly generated parity constraints.  From a theoretical viewpoint, approximation guarantees for XOR-MMAP are provided in both the unweighted (or binary) case and the weighted case. Comparative experiments, with respect to two state-of-the-art MMAP approximation methods are performed on various domains, including random SAT instances, an image completion task, and a network design application.
There are pros and cons in this article. On the one hand, the idea of approximating complex MMAP queries using a series of random constrained optimization problems is conceptually simple and opens the door to many potential improvements. Furthermore, I really appreciated the diversity of experimental domains used to evaluate the method. On the other hand, I have several concerns about the approximation bounds and the optimization oracle.  * Approximation bounds  The approximation guarantees provided by Theorem 3.2. and Theorem 3.6 are very weak. For the unweighted case, we have a bound of $4^c$ and, to ensure that the denominator $\alpha(c)$  of the complexity parameter T is greater or equal than 1, $c$ must be greater or equal than 5, which roughly means an approximation constant about one thousand. For the weighted case, the constant is about three thousands. With such large constants, one may wonder whether the approximation guarantees of the algorithm are useful in practice.  The extension to the weighted case (Section 3.2) could be improved using a recent article by Chakraborty et al. (IJCAI 15) (From Weighted to Unweighted Model Counting). Indeed, since the weight function is encoded as a literal-weighted CNF (as specified by Lines 358-362 in the S.M.), it can be transformed into an equivalent unweighted (binary) CNF formula of polynomial size. By applying Theorem 2 of Chakraborty and Theorem 5.2, it seems that we can get a better bound for the weighted case of MMAP.    In Lines 178-180, the authors suggest to use binary search instead of linear search for the number of calls to the optimization oracle. This is clearly a preferable strategy, but I am wondering whether the approximation bounds are still preserved in this case. In fact, the proof of Theorem 3.2 (Lines 173-177) needs $n$ calls to the oracle in order to derive a probability of $1 - \delta$. By the way, XOR-MMAP+ provided in the S.M. looks more interesting for reducing the complexity parameter $T$, but the number of calls to the oracle is in $O(n log n)$. So, can you obtain similar approximation bounds when XOR-MMAP(+) calls XOR-K only $O(log n)$ times?  * Optimization oracle  The XOR-MMAP algorithm relies on an optimization oracle which is far from easy to solve. Namely, Eq. (2) of Algorithm 2 is a nonlinear constrained optimization task, where the constraint set is a list of $kT$ random XOR clauses (Lines 87-88), each of size $O(n)$. Based on the standard encoding suggested in the supplementary material, the objective function can be specified as a linear function. But the overall set of constraints for this linear optimization problem involves both XOR clauses (encoding the hash function), and CNF clauses (encoding the weight function). As we know, XOR clauses over $n$ variables cannot be translated into CNF clauses (this would give an exponential number of disjuncts). So, which algorithm do you use to solve this linear optimization problem involving both CNF and XOR clauses? Unless I missed something, there is no discussion in the paper about the practical implementation of this expensive optimization oracle.      In light of the above remark, I think that it is crucial to report the running times of all algorithms in the experimental study. Even if XOR-MMAP behaves well in practice for approximating MMAP instances, its optimization oracle is very expensive. So one might really wonder whether XOR-MMAP is scalable, in comparison with SAA and Mixed LBP.   * Minor comments  Line 182: From Theorem 3.2 -> From Theorem 3.1  The use of “NP oracles” in the title is not - strictly speaking - appropriate. The procedure used by XOR-MMAP is an FP^NP (i.e. constrained optimization) oracle.


A Bandit Framework for Strategic Regression
We consider a learner's problem of acquiring data dynamically for training a regression model, where the training data are collected from strategic data sources. A fundamental challenge is to incentivize data holders to exert effort to improve the quality of their reported data, despite that the quality is not directly verifiable by the learner. In this work, we study a dynamic data acquisition process where data holders can contribute multiple times. Using a bandit framework, we leverage the long-term incentive of future job opportunities to incentivize high-quality contributions. We propose a Strategic Regression-Upper Confidence Bound (SR-UCB) framework, a UCB-style index combined with a simple payment rule, where the index of a worker approximates the quality of his past contributions and is used by the learner to determine whether the worker receives future work. For linear regression and a certain family of non-linear regression problems, we show that SR-UCB enables an O log T

This paper studies how to incentivize (self-interested) "arms" to exert effort to improve the quality of their predictions in a MAB setting. The setting assumes that no "ground truth" costs are ever revealed, giving  the problem an additional technical complication of how to measure the quality of the arms' predictions. They show in certain settings that there exist payment schemes and learning algorithms which have approximate BNE with arms exerting enough effort to give the algorithms nontrivial regret guarantees. They also show how to implement the updates of such algorithms in computationally less expensive way, and also in ways which preserve privacy of the arms.
 The question studied in the paper is interesting, and borrowing the idea from peer prediction to use the other arms' predictions as an unbiased estimator of the quality of one arm's prediction is a nice idea (in particular, because those arms need to be incentivized enough to make reasonably accurate predictions).   However,  the paper focuses too much on presenting "bells and whistles" rather than giving a deeper understanding of the basic (and main) results. Perhaps reorganizing the paper to only briefly mention the computational/privacy aware variants and giving both more intuition and technical content describing the main result (namely, that there exist \alpha-BNE with small regret) would focus the paper and give the reader a cleaner message of what the paper is doing.  This tact would have the added benefit that the reader might be able to better assess the "quantitative" consequences of this work, in that it would leave more room for the authors to ruminate on how much better or worse these bounds are than what one could get in the non-strategic setting, or in various trivial simplifications/special cases of this model. As the paper stands, this reviewer finds it difficult to assess from the main body of the paper alone the technical contribution of the paper (and whether the results follow from a mild reworking of standard proofs or need substantial, new ideas). It is also difficult to assess a theory paper which gives not even a sketch or an outline of a proof in the main body of the paper.  It would also be interesting to have some idea of what reasoning goes into designing the payment rules, or some lower bounds, or some impossibility results showing what sorts of BNE cannot be ruled out by any (reasonable) payment rule; since this is not a "PoA" style result that shows *all* apx-BNE have these regret guarantees, only that some will.  On a smaller scale, the paper has many typos that a spellchecker would catch, and many grammatical errors which make reading the paper cumbersome. This feels like a rough draft that a few days of love would substantially improve in this regard.  It also seems  somewhat strange that the assumptions about various parameters are scattered throughout the body of the paper; they should all be in one place or at least in the theorem statement(s).  Odds-n-ends  - run a spellcheck -"nowadays"? - "degrade the machine learning system's" -> "degrade machine learning systems' " - add "many" before "stable"? - "leveraged" should not be followed by "on" - "indexes" should be "indices" - "computationally light" is a weird phrase. - "the provided incentive much". What? Perhaps "without substantially degrading the incentives guarantee" - "beings" -> "being" - "after receiving the task" should be "after receiving a task" - worth mentioning that different "effort" functions could correspond to different levels of skill in workers - worth mentioning that the per-round payment scheme is "individually rational" - plurality seems to be a regular issue, as well as missing/incorrect articles (the, an, a, etc)


Optimal Architectures in a Solvable Model of Deep Networks
Deep neural networks have received a considerable attention due to the success of their training for real world machine learning applications. They are also of great interest to the understanding of sensory processing in cortical sensory hierarchies. The purpose of this work is to advance our theoretical understanding of the computational benefits of these architectures. Using a simple model of clustered noisy inputs and a simple learning rule, we provide analytically derived recursion relations describing the propagation of the signals along the deep network. By analysis of these equations, and defining performance measures, we show that these model networks have optimal depths. We further explore the dependence of the optimal architecture on the system parameters.

This paper describes a particular instantiation of a deep neural network with solvable dynamics during feedforward propagation of an input. They show for such an architecture, an optimal depth (number of layers) can be computed and determine analytically situations where higher depth is beneficial.
While this work attempts to elucidate a theoretical understanding of deep networks, the model they analyze diverges substantially from models which have been successful. For example, the weights in their model are 'trained' analytically in a layer-wise manner (via pseudo-inverse rule). Furthermore, the authors presume the function of each layer is to preserve the fundamental clustering structure of the input while progressively removing noise. There is no citation or work provided showing this empirically occurs in deep networks.  Unlike [1] which they cite as related theoretical work on deep learning, the do not connect their findings back to the models which are currently successful. Therefore, it is unclear how to connect their theoretical findings with the current field of deep learning.  That aside, the analysis and results do have close connections to hebbian learning and more connections should be made with that literature. Otherwise, the current connection to modern deep networks made in this work is tenuous.  [1] Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear 274 dynamics of learning in deep linear neural networks. arXiv.org, December 2013.


Natural-Parameter Networks: A Class of Probabilistic Neural Networks
Neural networks (NN) have achieved state-of-the-art performance in various applications. Unfortunately in applications where training data is insufficient, they are often prone to overfitting. One effective way to alleviate this problem is to exploit the Bayesian approach by using Bayesian neural networks (BNN). Another shortcoming of NN is the lack of flexibility to customize different distributions for the weights and neurons according to the data, as is often done in probabilistic graphical models. To address these problems, we propose a class of probabilistic neural networks, dubbed natural-parameter networks (NPN), as a novel and lightweight Bayesian treatment of NN. NPN allows the usage of arbitrary exponential-family distributions to model the weights and neurons. Different from traditional NN and BNN, NPN takes distributions as input and goes through layers of transformation before producing distributions to match the target output distributions. As a Bayesian treatment, efficient backpropagation (BP) is performed to learn the natural parameters for the distributions over both the weights and neurons. The output distributions of each layer, as byproducts, may be used as second-order representations for the associated tasks such as link prediction. Experiments on real-world datasets show that NPN can achieve state-of-the-art performance.

The paper presents a new Bayesian treatment for probabilistic neural networks. The approach allows to specify and estimate distributions within the exponential family for all parameters of the model, as well as its inputs an outputs. Besides, the framework allows for multi-layer architectures and nonlinear activation functions, akin to standard neural networks. With their approach, the authors aim to alleviate two well-know problems of standard neural networks, namely, overfitting when data is insufficient and lack of flexibility in terms of available distributions for weights and neurons.  The authors formulate neural networks in terms of parameters' distributions rather than the parameters themselves, which in principle allow the model to capture a richer representation of the data by modeling uncertainty. For additional flexibility, nonlinear activation functions are defined for a variety of distributions within the exponential family.  Learning is done via back-propagation and the authors provide details for 3 instances of their framework, namely, gamma, Gaussian and Poisson.  Experiments focus on handwritten digits classification (MNIST) and second-order representation learning for text documents using bag-of-words representations.
My main concern with the paper, on a conceptual level, is that the authors claim to use a Bayesian treatment of neural networks however, the authors make no mention whatsoever to posterior/predictive distributions and the only mention to priors is reserved to Gaussian distributions in the context of regularization. Also, all parameters of the model (weights and neurons) are specified via fully factorized univariate distributions. How is this different from a straightforward mean-filed variational Bayes treatment of neural networks with moment matching for activation functions? One of the most appealing elements of Bayesian analysis is the ability of estimating posterior/predictive distributions to quantify uncertainty via capturing the covariance structure of the data, however such task is very difficult with fully factorized specifications as it is well-known from the variational inference literature.  In the experiments, it is not clear to me what is the added value of the approach presented by the authors for three reasons: i) it is difficult to evaluate the significance of the results in Table 3 without error bars. Besides, for Size = 100, although NPN outperforms Dropout, it is not entirely obvious that the latter is overfitting considerably more than the former. ii) Why are the results for unsupervised models evaluated in terms on (indirect measure) link prediction rather than (direct measure) say perplexities or marginal likelihood estimates? iii) One of the main points of the authors is that standard neural networks are limited in terms of distribution choices, yet the authors show that on MNIST a Gaussian model outperforms the Gamma model which in principle is better suited for non-negative data. Besides on the unsupervised task, the Gaussian model compares favorably against the Poisson model, but again results in Table 4 are difficult to interpret as they indirectly quantify the representational capabilities of the models being considered.    It would be nice to get some clarity in the following points:  - How are the parameters of the activation functions in Table 3 selected/tuned in practice.  - How good is the approximation in (5), from the supplementary material, it seems that (5) comes from a well-known result for the probit link function, however, no details are provided about the accuracy of the approximation.  - In (6), what prevents the variance, a_s, from becoming a negative number? and if that is the case how it is handled in practice?  - In the Gaussian NPN model, outputs, y, are assumed Gaussian with variance epsilon, how is epsilon selected and how it affects performance?   - For Poisson NPN, how are counts obtained from the (continuous) approximation proposed by the authors?  - The authors should avoid using the term calibration so liberally, as the authors do not provide results to directly ascertain calibration, instead results simply indicate correlation between uncertainty (variance) and accuracy.


CYCLADES: Conflict-free Asynchronous Machine Learning
We present CYCLADES, a general framework for parallelizing stochastic optimization algorithms in a shared memory setting. CYCLADES is asynchronous during model updates, and requires no memory locking mechanisms, similar to HOG-WILD!-type algorithms. Unlike HOGWILD!, CYCLADES introduces no conflicts during parallel execution, and offers a black-box analysis for provable speedups across a large family of algorithms. Due to its inherent cache locality and conflictfree nature, our multi-core implementation of CYCLADES consistently outperforms HOGWILD!-type algorithms on sufficiently sparse datasets, leading to up to 40% speedup gains compared to HOGWILD!, and up to 5⇥ gains over asynchronous implementations of variance reduction algorithms.

In this paper, the authors propose an enhancement to the HogWild! Framework, which is claimed to maintain serial equivalence (i.e., produces the same outcome as the serial algorithm). Experimental results demonstrate that the proposed method outperforms HogWild! on a few tasks.
1) I am afraid that the paper contains over-claims. The Cyclades method actually only conditionally achieves the serial equivalence according to Theorem 1. The conflict degree is highly related to the sparsity of the data, and the condition of Theorem 1 cannot be satisfied in many practical scenarios. For example, for deep learning and other dense applications, the allocation algorithm cannot resolve conflicts at all. Even for sparse applications, when the data block is large, or the delay among different local machines is large, the serial equivalence cannot be achieved either. 2) As for the technical contribution of the paper, it seems not that significant over Hogwild!. To some extent, Cyclades adds a data partitioning step to Hogwild, and this data partitioning is done by leveraging existing bipartite allocation algorithms. 3) The authors made some analysis on the speed up achieved by Cyclades. However, speed up is only a system measure, and from the machine learning perspective, we care more about the balance between accuracy and efficiency. The speed up measure is defined with respect to \epsilon objective, which is a little tricky. \epsilon is defines as the smallest error achievable by parallel algorithm (but not by the serial algorithm) – this is unfair to the serial algorithm. According to my experience, in some cases, the parallel algorithm cannot achieve the same smallest error as the serial algorithms do. So, if we define \epsilon as the smallest error by the series algorithm instead, the parallel algorithm will have no speed-up at all. 4) As for the baseline in the experiment, the authors only compare with HogWild!. Given that there have many so many parallelization mechanisms proposed for distributed machine learning these years, only comparing with Hogwild! cannot sufficiently demonstrate the value of the proposed approach. It is necessary to compare with more and stronger baselines.  In general, I am not convinced by the rebuttal. According to the authors, serial equivalence is theoretically guranteed, but to me it is guaranteed in a very trivial way. When the data are dense, in order to gurantee the serial equivalence, one may have to use one single core which will make no sense of parallelization. Only if the data is sparse, the partition algorithm can take effect. Furthermore, many parallelization algorithms design for distributed setting can be easily adapted to the shared-memory setting, therefore only comparing with HogWild! seems not convincing enough to me.


Higher-Order Factorization Machines
Factorization machines (FMs) are a supervised learning approach that can use second-order feature combinations even when the data is very high-dimensional. Unfortunately, despite increasing interest in FMs, there exists to date no efficient training algorithm for higher-order FMs (HOFMs). In this paper, we present the first generic yet efficient algorithms for training arbitrary-order HOFMs. We also present new variants of HOFMs with shared parameters, which greatly reduce model size and prediction times while maintaining similar accuracy. We demonstrate the proposed approaches on four different link prediction tasks.

The authors propose a learning algorithm for so called higher order FMs (HOFMs) based on ANOVA kernels and the use of HOFMs with shared parameters. A backpropagation like algorithm is used for training.
It is an interesting, reasoned and promising approach. But there a few issues which I would like to have clarified in the rebuttal to accept the paper. 1. The idea of the paper seems to strongly rely on the paper "Polynomial Networks and Factorization Machines: New Insights and Efficient Training Algorithms" by Blondel et al., where ANOVA kernels have already been used. Can you explain in more detail the difference and contributions in comparison to this paper? 2. There has been extensive work on tensor factorizations which I am missing here. E.g., many papers by Tamara Kolda. I'm wondering why such approaches cannot be applied to the given problem or why it is not better to adapt them to HOFMs. Such comparisons to state-of-the art tensor factorizations are accordingly also missing in the experiments. Are there reason for this? If so, please highlight them. 3. The experiments are not very extensive. There is just one quite specific setting based on the prediction of presence or absence of links in graphs which isn't explained in much detail. Moreover, your method does not consistently perform better than the baselines. Not just for GD, also the performance differences for the other data sets are low. For NIPS the low-rank bilinear regression comes close to the value attained by a HOFM. For Enzymes and Movielens a Polynomial Network comes very close. It should be noted that the values vary quite a lot for different orders m, even in a nonmonotonic way. For example, the difference between a Polynomial Network with m=3 and both m=2, m=4 is bigger than the difference between a Polynomial Network with m=2 (or m=4) and the best value for a HOFM. This makes me wonder whether those values didn't just happen by chance. You also seem to have tried more settings for HOFMs than for the baselines, so it could be man-made overfitting. Also the data sets are rather small which could raise scalability questions. More detailed experiments would be better in my opinion. At least these issues should be explained.


Incremental Variational Sparse Gaussian Process Regression
Recent work on scaling up Gaussian process regression (GPR) to large datasets has primarily focused on sparse GPR, which leverages a small set of basis functions to approximate the full Gaussian process during inference. However, the majority of these approaches are batch methods that operate on the entire training dataset at once, precluding the use of datasets that are streaming or too large to fit into memory. Although previous work has considered incrementally solving variational sparse GPR, most algorithms fail to update the basis functions and therefore perform suboptimally. We propose a novel incremental learning algorithm for variational sparse GPR based on stochastic mirror ascent of probability densities in reproducing kernel Hilbert space. This new formulation allows our algorithm to update basis functions online in accordance with the manifold structure of probability densities for fast convergence. We conduct several experiments and show that our proposed approach achieves better empirical performance in terms of prediction error than the recent state-of-the-art incremental solutions to variational sparse GPR.

The author propose a novel variational approximation for Gaussian Process regression, by parametrizing the approximate posterior with a set of inducing functions, rather than inducing points. This approach makes on-line learning more feasible, because the the inducing points in previously used approximations are usually a subset of the data, which would not be available in an on-line setting. The approach is developed by using the dual representation of a GP in a RKHS. The exposition is mostly understandable, and the results are justifying the effort.
This is a very interesting, well written paper that tackles a relevant topic for the NIPS audience. Minor suggestions for improvement below:  - line 86, eqn.4: I assume f_{\tilde{X}} and f_X are the function values at the inducing points and the training data? please define. - line 110: "...whereas p(\tilde{X})..." do you mean p(f_{\tilde{X}) ? - line 126: "...of minimizing (8)..." do you mean maximizing? - lines 149-150: "The mean and the covariance functions...can be equivalently represented as a Gaussian distribution." this sentence made no sense to me. How can a distribution represent two functions? - line 156: "...such that \Sigma \geq 0..." should this be \tilde{\Sigma}? - line 172, eqn. (12): can the derivative of this expression, paricularly of the square root of the difference of determinants, lead to numerical instability when the difference btw. the matrices gets small? Please comment. - line 179: "...without referring to the samples.." this point is so important that it would be helpful for the reader if you wrote down the sample-independent form of the posterior here, so one does not have to piece it together from eqns 10-11 and the explanations that around these equations. line 188: "...with error \epsilon_t..." please define what this error is. line 191, eqn 16: the step from the second to the third line is puzzling, and the reader has to work through a whole paragraph before s/he is referred to the appendix where you explain the step.  Please put that reference closer to where you need it. - line 244: "...inducing functions to 512": ho did you come up with this number? what happens if you use less or more? Please also write down the inducing functions.  Proofs: - line 348: This derivation requires a number of auxiliary results which are proven afterwards. This is confusing for the reader. Please prove the auxiliary results first. Also, I fould the step from line 3 to 4, and from line 5 to 6 could do with more explanation. - line 349: please explain how you arrive at the first identity. - line 350: please explain how you arrive at the first identity. - line 354: second line of equations. please explain why the numerator can be rewritten like the denominator.  - line 354: last line of equations. I think there should be no p(f_X|f_{\tilde{X}) in the numerator. - line 367: what is \bar{\mu}?


Adaptive Averaging in Accelerated Descent Dynamics
We study accelerated descent dynamics for constrained convex optimization. This dynamics can be described naturally as a coupling of a dual variable accumulating gradients at a given rate η(t), and a primal variable obtained as the weighted average of the mirrored dual trajectory, with weights w(t). Using a Lyapunov argument, we give sufficient conditions on η and w to achieve a desired convergence rate. As an example, we show that the replicator dynamics (an example of mirror descent on the simplex) can be accelerated using a simple averaging scheme. We then propose an adaptive averaging heuristic which adaptively computes the weights to speed up the decrease of the Lyapunov function. We provide guarantees on adaptive averaging in continuous-time, prove that it preserves the quadratic convergence rate of accelerated first-order methods in discrete-time, and give numerical experiments to compare it with existing heuristics, such as adaptive restarting. The experiments indicate that adaptive averaging performs at least as well as adaptive restarting, with significant improvements in some cases.

Building upon an early work that connects accelerated Mirror Descent and continuous-time ODE, the authors extend the ODE dynamics with a more general averaging scheme and propose a novel adaptive averaging strategy for one of its discretization. In some cases, the new variant of AMD with adaptive averaging outperforms existing AMD algorithms with restarting.
The proposed adaptive averaging strategy is interestingly new and comes with a good intuition. While the paper looks nice overall, the originality and usefulness remains a big concern.   - Most elements (e.g. Lyapunov function, discretization, etc.)  directly follow from an early NIPS paper [7]. The extension from simple averaging to generalized averaging is kind of trivial and not of much use since eventually the authors only consider the case with quadratic rate; faster rate would require higher-order methods which are not always applicable.    - Some results, e.g.,  interpretation of energy function, reformulation into primal form, reduction to replicator dynamics, promote a better understanding of accelerated Mirror Descent algorithm, but are not significantly important to advance machine learning.   - Although the  adaptive averaging is introduced to satisfy the conditions sufficient to guarantee convergence of ODE, it is not fully theoretically grounded due to the approximation during discretization. Still, there is no explicit convergent result provided, meaning it is again just  a heuristic, as referred by the authors. Also, there exist a number of different averaging schemes of Nester's accelerated algorithms; without comparing to other alternatives, it is unclear whether this adaptive variant makes a big different.   - The experiment on a toy example with only quadratic/linear objective is not interesting or convincing enough to demonstrate the adavantage with adaptive averaging, especially for the general case with strongly convexity.     == post-rebuttal answer== I have read the authors' rebuttal and they promise to add - additional higher dimensional and non-quadratic examples in the revision - theoretical proof of convergence of adaptive averaging - implementation of adaptive averaging on Nesterov's cubic-regulariezed Newton method - discussion comparing to other adaptive averaging   I have updated my scores accordingly.


Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes
Neural networks augmented with external memory have the ability to learn algorithmic solutions to complex tasks. These models appear promising for applications such as language modeling and machine translation. However, they scale poorly in both space and time as the amount of memory grows -limiting their applicability to real-world domains. Here, we present an end-to-end differentiable memory access scheme, which we call Sparse Access Memory (SAM), that retains the representational power of the original approaches whilst training efficiently with very large memories. We show that SAM achieves asymptotic lower bounds in space and time complexity, and find that an implementation runs 1,000⇥ faster and with 3,000⇥ less physical memory than non-sparse models. SAM learns with comparable data efficiency to existing models on a range of synthetic tasks and one-shot Omniglot character recognition, and can scale to tasks requiring 100,000s of time steps and memories. As well, we show how our approach can be adapted for models that maintain temporal associations between memories, as with the recently introduced Differentiable Neural Computer.

The authors use fast approximate-nearest-neighbors structures and arg-top-K operations for computing the attention weights in an Neural Turing Machine.  They describe experiments showing that this does not degrade training performance, and in fact allows training on larger problems.
The novelty in this work is thin, see e.g. section 3.3. in the memory networks paper http://arxiv.org/pdf/1410.3916.pdf; there are now several memory network papers that use memories with millions of items and hashing methods to make the lookups scalable.    On the other hand,  I have not seen such tools used for writeable memories, and even if there is not a conceptual leap in this paper, certainly the practical aspects of making it work are worth reporting.   Nevertheless, I do recommend accepting the paper, but would like to see some changes.  1: I think it would be quite difficult to replicate the results of this paper from the descriptions in the paper.  Especially the descriptions of the tasks are lacking; considering the code for the tasks from the original NTM paper have not been released, and there is no "standard" version of the tasks, it is crucial that the paper give careful descriptions of the construction of the tasks.   I would also ask that the authors  commit to releasing the code for their experiments.     2: Where will this fail? does it always work? The authors often discuss their model as being smooth, but note that the argmax (or K-argmax) operation is *not* smooth.   With $K=1$, in the read only setting, their models are much like the MemNN-WSH in  in http://arxiv.org/pdf/1503.08895.pdf (although that model does not use hashing), and that model was reported to work less well than the fully smooth model.  Is it because of the $K=1$?  Because of the task?  I think this paper would be more valuable if there was some analysis of failure cases and some discussion of what kinds of tasks/setups allow their training to work: their model is making a discrete action, and they train without taking into consideration (as opposed to http://arxiv.org/abs/1511.07275).  I think that they get results in this way is a good contribution, but it would be better to understand what are the limits.  Also, I would suggest that the authors remove section 3.6, and I would also suggest the authors not call the omniglot data "non-synthetic" or "real-world".


A Non-convex One-Pass Framework for Generalized Factorization Machine and Rank-One Matrix Sensing
We develop an efficient alternating framework for learning a generalized version of Factorization Machine (gFM) on steaming data with provable guarantees. When the instances are sampled from d dimensional random Gaussian vectors and the target second order coefficient matrix in gFM is of rank k, our algorithm converges linearly, achieves O( ) recovery error after retrieving O(k 3 d log(1/ )) training instances, consumes O(kd) memory in one-pass of dataset and only requires matrixvector product operations in each iteration. The key ingredient of our framework is a construction of an estimation sequence endowed with a so-called Conditionally Independent RIP condition (CI-RIP). As special cases of gFM, our framework can be applied to symmetric or asymmetric rank-one matrix sensing problems, such as inductive matrix completion and phase retrieval.

This paper proposes a new algorithm for training a variant of Factorization Machines (FMs) (second-order polynomial parameterized by a low-rank matrix).  Pros:  - simple, efficient algorithm - novel convergence/recovery analysis *assuming* Gaussian-distributed data  Con: - no experimental validation at all  Note that the model considered in this paper is actually not the original FMs but a variant (some authors call it "polynomial networks"; see below). The authors might want to update the title and main body of their paper to reflect this.
Major comments --------------  * An obvious major issue with this paper is the lack of experiments. How does   the algorithm compare to gradient-based local search algorithms? I would be curious to see if it works better on i) Gaussian distributed data and ii) real data. Even if the results turn out to be similar, perhaps the authors can find some advantages to their algorithm such as, e.g., robustness to initialization.  * Technically, the model considered in this paper is a *variant* of FMs in two ways  1) The model considered uses the quadratic form x^T M x = \sum_{i=1}^d \sum_{j=1}^d M_{i,j} x_i x_j while FMs use \sum_{i=1}^{d-1} \sum_{j=i+1}^d M_{i,j} x_i x_j. Recent works [1, 2] have called this variant "polynomial network" so the authors might want to use this name instead of "factorization machine".  If the proposed algorithm and analysis trivially extends to the original FMs, the authors might want to add a few words about this as well.  [1] https://arxiv.org/abs/1410.1141  [2] http://jmlr.org/proceedings/papers/v48/blondel16.html  2) The original FMs factor M as V V^T, hence M is p.s.d. It would be interesting to briefly discuss this point. In particular, it is not clear whether a p.s.d. constraint is needed or whether it can hurt performance.  * Theorem 1 assumes that the data is Gaussian distributed. This   should be stated clearly in the abstract and introduction too.  Detailed comments -----------------  * Lines 80-81:  > When d is much larger than k, convex programming on the trace norm or nuclear norm of M∗ becomes intractable since M ∗ is of d × d.  This claim is wrong. Modern solvers for nuclear norm penalized minimization have no problem scaling to large d provided that the data is sparse. The authors should make a weaker claim or a more precise statement. For example, "although the estimation of M* can be formulated as convex programming based on the nuclear norm, a more popular strategy in practice is to ...".  * Lines 113-116:  You could also use \hat{x}_i^T M \hat{x}_i (i.e., use \hat{x}_i both on the left and right).  * How difficult is it to extend the proposed algorithm to other losses than the   squared loss?  * Line 123:   Due to the name "one-pass", I assume the instances are sampled without replacement?  * Algorithm 1:   H_3 seems to be a vector so it would be less confusing to not capitalize it.  * The precise objective function minimized in this paper does not seem to be   defined anywhere.  Typos -----  * Line 207: holds true -> hold true


An algorithm for 1 nearest neighbor search via monotonic embedding
Fast algorithms for nearest neighbor (NN) search have in large part focused on 2 distance. Here we develop an approach for 1 distance that begins with an explicit and exactly distance-preserving embedding of the points into 2 2 . We show how this can efficiently be combined with random-projection based methods for 2 NN search, such as locality-sensitive hashing (LSH) or random projection trees. We rigorously establish the correctness of the methodology and show by experimentation using LSH that it is competitive in practice with available alternatives.

The paper shows an embedding from l_1 into l_2-squared, for the application of nearest neighbor search. The embedding method can be adapted for a new out-of-sample extension (i.e., a previously unseen query point). The authors also show experiments showing the competitiveness of their method.
While the fundamental technical contribution is interesting and should be published, the contextualization is very far from ideal. Here are some points that the authors should address: - besides the l_1 methods based on Cauchy distribution, there is also LSH methods directly for the l_1 space: just impose a randomly shifted grid (see, e.g., the description in the CACM article by Andoni & Indyk). - it is not clear than embedding l_1 into l_2^2 would be the best way to tackly the l_1 problem. In particular, if we want to solve a c-approximation under l_1, this would translate into requiring a \sqrt{c} approximation for l_2 after the embedding. I.e., the resulting problem in l_2 is harder! Luckily, in l_2, LSH achieves runtime n^{1/c^2} (cf, l_1 achieves runtime n^{1/c}); however the corresponding l_2 algorithns are much harder (compare the algorithms from  the reference [10] versus [5]). Thus, it seems more beneficial to tackle l_1 problem directly. It is possible that, in practice, l_2 algorithms are much more performant, but this need to be argued well.  The authors should mention also that an embedding from l_1 into l_2-squared is actually known from [Linial, London, Rabunovich, Combinatorica 1995]. That embedding proceeds by embedding l_1 into Hamming space using unary embedding, and then embedding Hamming into l_2-squared using identity mapping. Its disadvantage is that the "scale" needs to be known to achieve a bounded dimension (which is a common assumption in LSH in fact).  The algorithm from this paper in fact can be seen as an efficient application of the JL dimension reduction on the embedding resulting from the LLR construction. This is what I find the most interesting contribution in this paper, and it should be played up. However, I cannot recommend a positive score due to the issues from above.


Combining Adversarial Guarantees and Stochastic Fast Rates in Online Learning
We consider online learning algorithms that guarantee worst-case regret rates in adversarial environments (so they can be deployed safely and will perform robustly), yet adapt optimally to favorable stochastic environments (so they will perform well in a variety of settings of practical importance). We quantify the friendliness of stochastic environments by means of the well-known Bernstein (a.k.a. generalized Tsybakov margin) condition. For two recent algorithms (Squint for the Hedge setting and MetaGrad for online convex optimization) we show that the particular form of their data-dependent individual-sequence regret guarantees implies that they adapt automatically to the Bernstein parameters of the stochastic environment. We prove that these algorithms attain fast rates in their respective settings both in expectation and with high probability.

The paper continues a line of research getting more and more popular in online learning (and learning theory in general): how to design algorithms that perform well in the adversarial setting yet are able to utilize additional regularities in the data that make the task at hand easy. It is shown that some existing algorithms in online learning (Squint and MetaGrad) are able to adapt to easy stochastic environments satisfying the Bernstein condition (used in proving fast rates in stochastic loss minimization), while keeping the usual adversarial guarantees.
ADDED AFTER THE RESPONSE:  I am happy with the response and support the acceptance of the paper. Please include the discussions from the rebuttal to the paper.   Two minor things:  - I still think that unifying the notation is possible (in the expert case you have to restrict U to the corners), but I might be missing something here. - Bernstein to Central: Application of the newly cited lemma is ok, but mention the required boundedness conditions (|2\eta x^f| \le 1.79... which requires D,G-dependent assumptions on eta for the OCO case.  ==============================================  In general I find the problem important and interesting and the treatment elegant. The derivation of the results do not seem to require new techniques, although certainly there are some interesting technicalities in the proofs. While the presentation is good, it could be improved: I think that related work should be discussed a bit differently, and the organization of the paper can be improved to put the results more in context and emphasize on the more relevant parts (these can be done easily, suggestions are given below).  First of all, a slightly weakened version of the intended adaptive bounds can be achieved trivially by applying either the AB-prod algorithm of Sani et al (2014) or the algorithm of the much earlier paper of Even-Dar et al (Machine Learning 2008, "Regret to the best vs. regret to the average, this should also be cited) for an adversarial learning algorithm and the an algorithm that can adapt to the Bernstein condition in the stochastic case (e.g., Bartlet and Mendelson, 2006). You should argue how and why your results are better than this (which is not hard). (This argument also applies as a solution to the "linear worst case regret" in l. 169.)  The results essentially apply to Squint and MetaGrad, but these algorithms are not provided. While Squint is at least published, MetaGrad is only available as a preprint (and by no means is so widely known that it should not be included).   The paper continuously derives results for Squint and MetaGrad separately, from time to time introducing some unifying notation. It seems to me that treating the expert case (Hedge setting) as a problem of linear prediction over the simplex would automatically unify the two problems (the original losses will become gradients).  In the statements of the main results (Theorem, 3 Theorem 9), why do you recall MetaGrad and Squint? Wouldn't the main statement be that given data dependent regret bounds fast rates hold for the Bernstein case? By the way, wouldn't your results apply for Adapt-ML-Prod (maybe I am missing something here)?   * Organization: I think that Lemma 7 (and the related notation) adds very little to understanding the proof of the theorem. Especially, the trivial proofs should be relegated to the appendix. This could free some space to provide more important materials (e.g., more discussion, high level proof steps of Theorem 9, or the proof of Theorem 3).   * Technical questions: 1. In the problem definition, make it clear if the environment is oblivious or not. While in general your results apply to non-oblivious environments, conditional expectations (e.g., in Sec 2.3) are not taken according to that.  2. Sec 2.3: Wouldn't it be enough to require that k^* minimizes the expected cumulative loss?  3. Since the Hedge setting is only defined for countably infinite expert sets, it is somewhat bothering to extend it to the undefined uncountable case. Maybe define the problem generally, then restrict it to the countable case when necessary?  4. Appendix B: It would be better to repeat the Bernstein Sandwich lemma (it is very hard to resolve the notation in the reference). Also, I am not sure that the cited lemma is correct: in the last step of its proof, both sides are multiplied by x^2, but why can you take its minimum in the upper bound? (There are too many parentheses in line 424.)  5. Theorem 13: I find the statement of the theorem a bit confusing. In particular, the O(K_T+C_T) notation is a bit fuzzy when nothing is assumed about these sequences. Why not make an assumption that K_T/T and C_T/T go to zero, in which case the bound would have the nice form of T O(((K_T+C_T)/T)^1/(2-kappa)). Also consider removing the intuitive explanations from mathematical statements (maybe this is just my taste). It is very hard to associate any meaning to this part of the theorem.   * Minor comments: - Introduction, par 2: There are several other types of results (combinatorial problems, context tree prediction, tracking, curvature of the loss function, etc) which adapt to some sort of problem complexity. Consider mentioning what sort of complexity you are after. - l. 66: The definition of \pi might not be entirely clear, in particular that the upper index is not power and that the distribution is over the natural numbers (it will be automatically if you define the algorithm). - l. 100: What do you mean by "the Bays act is in the model?" - l. 116: "these stochastic conditions" -> the following stochastic conditions.  - l. 122: only equivalent with appropriate choices of parameters. - l. 130: "Second order methods" seems to be a bit too general here. - l. 175-180 is not entirely clear (without reading the relevant part in the Appendix). Why is this discussion restricted to Squint? - l. 191-193: In fact you loose a log log T factor by relaxing the assumptions - l. 198: consider->considered - l. 218: "we model" should rather be "Assume z_1,...,z_T is a Markov chain" - Section 3.4: I find it a bit overstatement to call the considered linear loss Hinge loss (since the "hinge part" is excluded). In what sense is the maximum of the Hinge loss 1 (consider a large negative < x,u >)? - Lemma 7: Your statement in 1 is really not that X is negative with high probability, only that it is not too large. Also, the notation about the expectations is very confusing here (e.g., E_{f\sim w} only takes expectation over f (with all other randomness in X^f fixed, wile E does not affect f in E X^f. Please make these more precise/clear (define them; applies to other occurrences, e.g., the proof of Theorem 9). - l. 284: give a reference for the dual representation. - Theorem 9: remove "the" before "excess". gamma is not defined to be 2 eta. This is also needed in the proof (l. 434). - l. 312: 2 epsilon should be 2 gamma - The references are really hard to parse in the current format. - l. 435: You have to take the proper conditional expectations to be able to invoke the chain rule. - l. 455: Mention that the omitted kappa-dependent constant is at most 1. - Example 1: What is \mathcal{J}? Please make the notation consistent throughout. (correct "m utandis") - l. 459: Did you use an assumption that \gamma^2 \le 1 (to deal with (the 1+gamma^2) terms)?


Mutual information for symmetric rank-one matrix estimation: A proof of the replica formula
Factorizing low-rank matrices has many applications in machine learning and statistics. For probabilistic models in the Bayes optimal setting, a general expression for the mutual information has been proposed using heuristic statistical physics computations, and proven in few specific cases. Here, we show how to rigorously prove the conjectured formula for the symmetric rank-one case. This allows to express the minimal mean-square-error and to characterize the detectability phase transitions in a large set of estimation problems ranging from community detection to sparse PCA. We also show that for a large set of parameters, an iterative algorithm called approximate message-passing is Bayes optimal. There exists, however, a gap between what currently known polynomial algorithms can do and what is expected information theoretically. Additionally, the proof technique has an interest of its own and exploits three essential ingredients: the interpolation method introduced in statistical physics by Guerra, the analysis of the approximate message-passing algorithm and the theory of spatial coupling and threshold saturation in coding. Our approach is generic and applicable to other open problems in statistical estimation where heuristic statistical physics predictions are available.
Consider the following probabilistic rank-one matrix estimation problem: one has access to noisy observations w = (w ij ) n i,j=1 of the pair-wise product of the components of a vector s = (s 1 , . . . , s n ) ∈ R n with i.i.d components distributed as S i ∼ P 0 , i = 1, . . . , n. The entries of w are observed through a noisy element-wise (possibly non-linear) output probabilistic channel P out (w ij |s i s j / √ n). The goal is to estimate the vector s from w assuming that both P 0 and P out are known and independent of n (noise is symmetric so that w ij = w ji ). Many important problems in statistics and machine learning can be expressed in this way, such as sparse PCA [1], the Wigner spiked model [2,3], community detection [4] or matrix completion [5].
Proving a result initially derived by a heuristic method from statistical physics, we give an explicit expression for the mutual information (MI) and the information theoretic minimal mean-square-error (MMSE) in the asymptotic n → ∞ limit. Our results imply that for a large region of parameters, the posterior marginal expectations of the underlying signal components (often assumed intractable 30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.
Factorizing low-rank matrices has many applications in machine learning and statistics. For probabilistic models in the Bayes optimal setting, a general expression for the mutual information has been proposed using heuristic statistical physics computations, and proven in few specific cases. Here, we show how to rigorously prove the conjectured formula for the symmetric rank-one case. This allows to express the minimal mean-square-error and to characterize the detectability phase transitions in a large set of estimation problems ranging from community detection to sparse PCA. We also show that for a large set of parameters, an iterative algorithm called approximate message-passing is Bayes optimal. There exists, however, a gap between what currently known polynomial algorithms can do and what is expected information theoretically. Additionally, the proof technique has an interest of its own and exploits three essential ingredients: the interpolation method introduced in statistical physics by Guerra, the analysis of the approximate message-passing algorithm and the theory of spatial coupling and threshold saturation in coding. Our approach is generic and applicable to other open problems in statistical estimation where heuristic statistical physics predictions are available.
Consider the following probabilistic rank-one matrix estimation problem: one has access to noisy observations w = (w ij ) n i,j=1 of the pair-wise product of the components of a vector s = (s 1 , . . . , s n ) ∈ R n with i.i.d components distributed as S i ∼ P 0 , i = 1, . . . , n. The entries of w are observed through a noisy element-wise (possibly non-linear) output probabilistic channel P out (w ij |s i s j / √ n). The goal is to estimate the vector s from w assuming that both P 0 and P out are known and independent of n (noise is symmetric so that w ij = w ji ). Many important problems in statistics and machine learning can be expressed in this way, such as sparse PCA [1], the Wigner spiked model [2,3], community detection [4] or matrix completion [5].
to compute) can be obtained in the leading order in n using a polynomial-time algorithm called approximate message-passing (AMP) [6,3,4,7]. We also demonstrate the existence of a region where both AMP and spectral methods [8] fail to provide a good answer to the estimation problem, while it is nevertheless information theoretically possible to do so. We illustrate our theorems with examples and also briefly discuss the implications in terms of computational complexity.

The paper considers the estimation of a vector S by observing noisy versions W_{ij} of its coordinate-wise product terms s_i.s_j normalized by \sqrt{n}, under additive Gaussian noise of variance \Delta. The main result is a formula for the limit  (1) \lim_{n --> \infty} 1/n I(S; W)  in terms of the replica symmetric potential function i_{RS}(E, \Delta), which, as a corollary (using the I-MMSE relation), shows that MMSE(\Delta) for the estimation problem above is given by the minimizer E^* of i_{RS}(E, \Delta). As a further corollary it is shown that Approximate Message Passing (AMP), which is known to converge to a stationary point of i_{RS}(E, \Delta) when \Delta is in appropriate range, yields MMSE in the same range. The proof of (1) is the main contribution of the paper. The proof is based on establishing an equality of the two regions over an appropriate interval and then extending it to a larger region over which the (complex extensions of the) two sides are analytic. A key step is to show that the analytic threshold for the left-side of (1), termed \Delta_{OPT}, is at least as much as that for the right-side, \Delta_{RS}. This is facilitated by relating the two thresholds to the threshold for a spatially coupled construction of W_{ij} (which is done in Lemma 3.1 and Lemma 3.2).
The paper addresses an important topic, namely that of providing formal guarantees for algorithms derived from heuristics in statistical physics. The approach proposed in the paper is very interesting and perhaps novel, at to this reviewer it is, wherein the gap in the thresholds for information theoretic performance and the performance of AMP is bridged by connecting the information theoretic threshold and that for AMP to that of a spatially coupled model. This relation in turn relies on the equality of the threshold for spatially coupled construction and the optimal information theoretic threshold, a phenomenon that has led to a recent breakthrough capacity achieving construction in information theory and is proved here again via Lemma 3.1. Another key step is to show that \Delta_{RS} does not exceed the threshold for the spatially coupled  construction with AMP; this is  the content of Lemma 3.2. While only a sketch of proof is presented, the approach looks convincing and indeed very interesting. The analysis proposed in the paper, if rigorously correct, leads to an interest approach for analyzing the performance of AMP by bringing in a spatially coupled system to relate various thresholds. Whether this contribution is new is beyond the knowledge of this reviewer. Nevertheless, this overall approach is very fascinating.   In addition to the general comments above, I have the following specific comments:  1. The paper and the contribution is rather difficult to follow. The authors mention the connection of their results to several problems  but then only present two applications via examples in Section 2 without clarifying the exact implication of their approach for these examples. In particular, what was the state-of-the-art before this paper and how has this paper furthered the analysis of these applications? Also, I suggest that the authors spend some more space in explaining the connection between the formula derived here and the general inference problem. Perhaps the relation of the left-side to information theoretic threshold and the right-side to the threshold of AMP (via the replica symmetric potential function) can be mentioned at the outset without much details.  2. The proof sketch is very difficult to follow. The authors should first summarize the plan before getting to the details. I have written above what I could follow, which might be wrong. However, a similar description in words of the proof idea of the authors will be very useful.   3. The sentence leading to (8) and the one following it are unclear to me.  In particular, I couldn't follow the observation "so the left hand side of (7) is equal to the derivative of \min_{E}i_{RS(E, \Delta) w.r.t. \Delta^{-1}". Does the other term in i_{RS}(E, \Delta) equals 0 at the min? Also, in the sentence after (8), it seems you interchanged the limit and the integral. If you justify this interchange in a complete version, at least point-out in the sketch that it needs to be done. Finally, understanding the proof sketch of Lemma 3.2 seems beyond the scope of a non-expert.   4. The extensions claimed in 135-39 seem to be much more interesting than the simple case here. If the authors indeed have these extensions available, why not motivate the paper by these extended results. If such extensions are not available as of now, please modify the claim above to be clear about it.


Reshaped Wirtinger Flow for Solving Quadratic System of Equations
We study the problem of recovering a vector x ∈ R n from its magnitude measurements y i = | a i , x |, i = 1, ..., m. Our work is along the line of the Wirtinger flow (WF) approach Candès et al. [2015], which solves the problem by minimizing a nonconvex loss function via a gradient algorithm and can be shown to converge to a global optimal point under good initialization. In contrast to the smooth loss function used in WF, we adopt a nonsmooth but lower-order loss function, and design a gradient-like algorithm (referred to as reshaped-WF). We show that for random Gaussian measurements, reshaped-WF enjoys geometric convergence to a global optimal point as long as the number m of measurements is at the order of O(n), where n is the dimension of the unknown x. This improves the sample complexity of WF, and achieves the same sample complexity as truncated-WF Chen and Candes [2015]  but without truncation at gradient step. Furthermore, reshaped-WF costs less computationally than WF, and runs faster numerically than both WF and truncated-WF. Bypassing higher-order variables in the loss function and truncations in the gradient loop, analysis of reshaped-WF is simplified.

The authors propose a method for phase retrieval which is different with WF (a previous algorithm for the same problem) in the function, and uses a non smooth one.They achieve measurement complexity nearly equal to what we have in the previously proposed method which is a truncated version of WF or TWF, and hence the sample complexity is improved in comparison with WF. Since the function is non smooth we do not have gradient here and the term gradient-like refers to what they define as gradient. The initialization is like TWF with a small change in truncation bounds which is because of removing power of two of the measurements. The gradient-like step is more close to WF since they do not use truncation in this part but they remove those points that gradient is equal to zero and claim that these points are few. There is a comparison of initialization which shows their algorithm works better. They illustrate the difference between the loss functions of their method and WF to give an intuition to reader about why their method is faster. Also there is results of the comparison between new method, WF and TWF for CDP model, Complex case and real case which shows that in complex case and CDP model the new method works better than the two others. The other experiment is for iteration count and time complexity for n=1000 and m=8n which they added Altminphase algorithm to comparison as well. For the comparison of time and iteration they also state time and iteration count in a table for image of galaxy with CDP model and L=16 and showed the recovered image with new method. Finally there is a comparison for poisson case. They proved geometric convergence and performance guarantee of the initialization and also have expectation of loss function.
The reviewer kindly suggests to enlarge some of the plots and to add comparison for different values of m and n, for example smaller values and also show the errors in the tables. Also suggests adding figures of other recovered images of other algorithms with the values of error and for smaller values of L to make a better comparison. The reviewer thinks it is interesting to see how the new method is working for small values of measurements or L in comparison to previous methods since L=16 is very large. It may be confusing that why the authours insist on real case while their algorithm works worse than TWF in this case and needs more measurements. If using the real case is just for simplicity of the proofs, it may be better to be explained and even just be used for the proof section. It may also cause confusion to use term gradient when there is no gradient actually and it is better to use the original word. Also it may be of great interest to add the reason behind better performance of initialization step in comparison with TWF.


Estimating Nonlinear Neural Response Functions using GP Priors and Kronecker Methods
Jointly characterizing neural responses in terms of several external variables promises novel insights into circuit function, but remains computationally prohibitive in practice. Here we use gaussian process (GP) priors and exploit recent advances in fast GP inference and learning based on Kronecker methods, to efficiently estimate multidimensional nonlinear tuning functions. Our estimator requires considerably less data than traditional methods and further provides principled uncertainty estimates. We apply these tools to hippocampal recordings during open field exploration and use them to characterize the joint dependence of CA1 responses on the position of the animal and several other variables, including the animal's speed, direction of motion, and network oscillations. Our results provide an unprecedentedly detailed quantification of the tuning of hippocampal neurons. The model's generality suggests that our approach can be used to estimate neural response properties in other brain regions.

The authors make use of a recently developed and fast variational algorithm to learn Gaussian processes (GP) from data with non-Gaussian likelihoods to learn the tuning properties of neurons in hippocampal cells. After quickly introducing the mathematical details on which the main algorithm is based, the authors show the robustness of their estimators on simulated data. Finally, they apply the technique on data from hippocampal area AC1 and discuss their findings.
In this work the authors make use of an already developed technique, so for this reason I don't find the paper innovative on the technical side. The authors emphasize that the focus is on the application of the method to infer multi-dimensional tuning functions of hippocampal neurons (line 40). Nevertheless they don't try to infer tuning functions in more than 3 dimensions (2 spatial + 1 non-spatial variables), and in my opinion this reduces the potential novelty that their results may have. In the next paragraphs I give a detailed opinion about the content and the structure of the paper.  Content  Subsection "Detailed parameter settings for experiments", page 4. The authors may consider to mainly reorganize this section, maybe also to remove it completely from here and merge it with the section in which they talk about the application of the method of neural data (see the following points in my review).   Section 3 "Results": In my opinion, there should be a clear separation between the estimator validation section (which may include all the experiments run on simulated data) and the section where the method is applied to data from neural recordings.  Subsection "Spectral mixture kernels for complex functional dependencies", page 5. I am not sure about the usefulness of the first experiment (extrapolating the tuning function outside rectangular region): the fact that using a spectral mixture kernel make it possible make extrapolations was already a well established fact in reference [9], and I would be surprise if this was not the case in this simple experiment. Rather, I would suggest to focus more on the second experiment, in which learning a correct representation of the tuning function is not trivial at all.  Subsection "Spatial and non-spatial modulation of CA1 responses": The authors may consider to reorganize the structure of this subsection, which represents the very core of the paper. A brief description of the data should be given at the very beginning of the paragraph (experimental techniques, number of recorder cells, few words about preprocessing the raw signals). I noticed that some of this information is already reported, but spread across the whole section, making it difficult for the reader to have a clear picture of what kind of data was used. An important analysis that in my opinion is missing in this sections is the estimation of the full joint posterior distribution over all the spatial and non-spatial variables at the same time. Only multiple 3D fields consisting of the spatial location plus an additional non-spatial variable are estimated. The authors claim that this choice was made for the sake of simplicity (Footnote #11), but the focus of the paper is "not on the methods per se but rather on their previously unacknowledged utility for estimating multidimensional nonlinear tuning functions", as stated by the authors (line 40). Therefore it is really important to see these methods applied to data which is really multidimensional (with dimension larger than 3). In this case, the 3D fields estimation would still be useful, using their predicted marginal posterior distributions to assess the quality of the the marginal posterior distributions predicted by the full multidimensional field.  General questions/remarks  I did not understand why the corners are missing in the plots in Figure 1d. It looks like this plots correspond to the walls of a box that was open and flatten on the plane, somehow corresponding to a 2D representation of a 3D plot. Is this the case? Please consider to specify it in the caption of the figure or to change the shape of the plot, because, as it is now, it is difficult to understand.  The authors should consider to reduce the number of footnotes. Overall, there are 14 footnotes, which sometimes contain also important information (#11, #12, #14).  Page 4, line 126: the authors refer to a parameter "k", stating that it takes discrete values. Nevertheless, the meaning of "k" is not defined until line 176.  Page 4, line 135: a reference would be important here.  On page 7: several correlations between pairs of variables are reported. In all cases, brackets are misaligned, and the last one is always missing (lines 207, 208, 211, 221, 222).


Near-Optimal Smoothing of Structured Conditional Probability Matrices
Utilizing the structure of a probabilistic model can significantly increase its learning speed. Motivated by several recent applications, in particular bigram models in language processing, we consider learning low-rank conditional probability matrices under expected KL-risk. This choice makes smoothing, that is the careful handling of low-probability elements, paramount. We derive an iterative algorithm that extends classical non-negative matrix factorization to naturally incorporate additive smoothing and prove that it converges to the stationary points of a penalized empirical risk. We then derive sample-complexity bounds for the global minimzer of the penalized risk and show that it is within a small factor of the optimal sample complexity. This framework generalizes to more sophisticated smoothing techniques, including absolute-discounting.

Given two discrete random variables X and Y (each taking k values), the paper considers the problem of estimating the k X k matrix P defined by P(i, j) = P(Y = j| X = i) from n i.i.d observations (X_s, Y_s), s = 1,..,n under the assumption that P can be factorized as AB for two similar stochastic matrices A and B of orders k X m and m X k respectively. The paper proposes an alternating optimization scheme for estimating P. A complication arises from the fact that many of the observed counts may be zero and the author propose at least two different ways of tackling it.
If my understanding is correct, Theorem 1 of the authors does not quite apply to their algorithm ADD-1/2-Smoothed Low-Rank. Instead, it applies to the non-computable algorithm where they assume that they have a minimizer of the objective function in Theorem 3. It is not clear if the alternating optimization algorithm proposed in the paper is guaranteed to converge to a minimizer of the objective in Theorem 3. If this is true, the authors should mention this before stating Theorem 1 to avoid misleading the reader.   The "discounting" seems important from the Experiments section but this is not described in the main paper. If this is so important, the authors should make room for this in the main paper.   The main results  (Theorem 1 and 2) are not so surprising given that this is almost a parametric estimation problem with ~mk parameters (so the rates should be km/n). The interesting aspect here is if there would be log terms in the minimax rate. Theorem 1 has log-factors while Theorem 2 has none. It will be great if the authors can determine if there should be a log term or not in the minimax rate.   It would also be helpful if the authors can comment on how to choose m somewhere in Section 5. It appears from their experiments that they do this via cross validation but a careful specification in Section 5 would be appreciated.


Online Bayesian Moment Matching for Topic Modeling with Unknown Number of Topics
Latent Dirichlet Allocation (LDA) is a very popular model for topic modeling as well as many other problems with latent groups. It is both simple and effective. When the number of topics (or latent groups) is unknown, the Hierarchical Dirichlet Process (HDP) provides an elegant non-parametric extension; however, it is a complex model and it is difficult to incorporate prior knowledge since the distribution over topics is implicit. We propose two new models that extend LDA in a simple and intuitive fashion by directly expressing a distribution over the number of topics. We also propose a new online Bayesian moment matching technique to learn the parameters and the number of topics of those models based on streaming data. The approach achieves higher log-likelihood than batch and online HDP with fixed hyperparameters on several corpora. The code is publicly available at https://github.com/whsu/bmm.

The paper aims to propose a simpler model than HDP which will explicitly control the number of topics. The paper is confusing in many parts and lacks minimum essential description to validate the contribution.
 Although the argument around simplicity appears multiple times in the paper but it is kept vague. I assume it means less number of random variables in the joint distribution of the model. However, line 19 indicates me that HDP is complex due to difficulty in understanding its theoretical structure.   Then it implies that the aim is to extend LDA generative process by sampling the number of topics.  The strategy of the method is to compute likelihood by varying number of topics from 1 to K and pick the one with maximum likelihood. This in some sense does the standard process of selecting number of topics using cross-validation i.e. learn a model with a value for number of topics and then check likelihood on some held-out data and pick the one with maximum likelihood. Here the validation and training sets are same.  But due to prior over T, the algorithm can provide some initial bias.   The concept is interesting, but initial paragraphs could create confusion. Some examples below.  (i) line 5-6, "it is a complex model and it is difficult to incorporate prior knowledge since the distribution over topics is implicit." -- next lines indicate that authors wanted to say the number of topics is not explicitly controlled in HDP, but these lines make it confusing.   (ii) "Uniform process may also be used to avoid priors with rich getting richer property" (line 26-27) -- this is correct but is misleading as the context is topic modeling and uniform process does not apply for grouped datasets. The mechanism to share support among multiple uniform processes is still unknown. Such statements should be carefully made.  In Eq. 1, the challenge is to compute c_n the normalization factor. In sec 4.1 variational approximation has been introduced, then corresponding sequential update equation to Eq. 1 should be made clear. When P_n is made as prior for (n+1)th sample as mentioned in line 147, then there is some difference with exact posterior which is not discussed here. In this context it is relevant to discuss Online Learning for Latent Dirichlet Allocation, Hoffman et al, NIPS 2010 and  Online variational inference for the hierarchical Dirichlet process, Wang et al, AISTATS 2011. Through moment matching the paper could possibly propose alternatives but the essential details have not been discussed here.   In Algorithm 1, value of \gamma depends on \alpha and \beta which in turn depend on \gamma,it is not clear how the values are computed for each sample. Currently it seems there is no iteration and they are initialized only once at the beginning that means truncation level for document n depends on other variables till document n-1. Then it confuses me how we are maximizing likelihood at each step.  This could be improved by discussing sequential update considering variational approximation and moment matching together. In that sense lines 153-154 is misleading. As this paper is about online method based on moment matching then the content lacks required details.   Authors should also note that the number of topics here is controlled by the choice of prior over T or initial value of \gamma, which is regarded as explicit control I think. It is important to discuss that similar behaviour is observed in HDP through the scale variable of DP. That controls the growth in number of topics. If the paper is about explicitness in control over number of topics, then this part requires some discussion.