Scalable Robust Matrix Factorization with Nonconvex Loss
Matrix factorization (MF), which uses the 2 -loss, and robust matrix factorization (RMF), which uses the 1 -loss, are sometimes not robust enough for outliers. Moreover, even the state-of-the-art RMF solver (RMF-MM) is slow and cannot utilize data sparsity. In this paper, we propose to improve robustness by using nonconvex loss functions. The resultant optimization problem is difficult. To improve efficiency and scalability, we use majorization-minimization (MM) and optimize the MM surrogate by using the accelerated proximal gradient algorithm on its dual problem. Data sparsity can also be exploited. The resultant algorithm has low time and space complexities, and is guaranteed to converge to a critical point. Extensive experiments show that it outperforms the state-of-the-art in terms of both accuracy and speed.
RMF solvers cannot be used. As in RMF-MM, we rely on the more flexible MM optimization technique, and a new MM surrogate is proposed. To improve scalability, we transform the surrogate to its dual and then solve it with the accelerated proximal gradient (APG) algorithm [2,32]. Data sparsity can also be exploited in the design of the APG algorithm. As for its convergence analysis, proof techniques in RMF-MM cannot be used as the loss is no longer convex. Instead, we develop new proof techniques based on the Clarke subdifferential [10], and show that convergence to a critical point can be guaranteed. Extensive experiments on both synthetic and real-world data sets demonstrate superiority of the proposed algorithm over the state-of-the-art in terms of both accuracy and scalability.
Notation. For scalar x, sign (x) = 1 if x > 0, 0 if x = 0, and −1 otherwise. For a vector x, Diag(x) constructs a diagonal matrix X with X ii = x i . For a matrix X, X F = ( i,j X 2 ij ) 1/2 is its Frobenius norm, X 1 = i,j |X ij | is its 1 -norm, and nnz(X) is the number of nonzero elements in X. For a square matrix X, tr(X) = i X ii is its trace. For two matrices X, Y , X Y denotes their element-wise product. For a smooth function f , ∇f is its gradient. For a convex f , G ∈ ∂f (X) = {U : f (Y ) ≥ f (X) + tr(U (Y − X))} is a subgradient.

In this paper, an algorithm for matrix factorization (MF) is proposed, which provide better robustness to outlier’s behavior compared to state of the art algorithms for Robust MF (RMF). Another nice property of the new algorithm is that it is suitable for sparse data matrices which is a clear advantage when it is compared against previous approaches to RMF. Basically, the authors propose to replace the l1 penalty with a nonconvex regularizer and provide results for various selections of regularizers. Since the optimization in this case is not simple, the authors propose to use a Majorization Minimization (MM) optimization approach with a definition of a new MM surrogate. The paper provides a nice introduction to the problem of robust matrix factorization, providing a complete overview of the state of the art and giving good motivations for the development of a new algorithm. I found the paper well written, technically sounded and having a nice structure. The theoretical treatment of the results is adequate giving guarantees of convergence. The paper includes a rich experimental section with useful comparisons with previously proposed algorithms and evaluating several options for regularizers. I definitively recommend the acceptance of this submission for presentation in NIPS. However, I found few minor issues which need attention for the final submission. Issues: • Font sizes in all figures are too small. Please consider resizing figures. • In section 4.1 Synthetic Data, outliers are generated by using constant values {+5, -5} in the 5% of the entries. I think, it would be very useful to analyze the effect of varying the percentage of affected entries (instead of a fixed 5%) and also to consider other values for outliers with some probabilistic distribution. I think that, by adding these new analysis the applicability of the algorithm would be better assessed. The authors agreed to add these additional results to the final version.


Algorithmic Assurance: An Active Approach to Algorithmic Testing using Bayesian Optimisation
We introduce algorithmic assurance, the problem of testing whether machine learning algorithms are conforming to their intended design goal. We address this problem by proposing an efficient framework for algorithmic testing. To provide assurance, we need to efficiently discover scenarios where an algorithm decision deviates maximally from its intended gold standard. We mathematically formulate this task as an optimisation problem of an expensive, black-box function. We use an active learning approach based on Bayesian optimisation to solve this optimisation problem. We extend this framework to algorithms with vector-valued outputs by making appropriate modification in Bayesian optimisation via the EXP3 algorithm. We theoretically analyse our methods for convergence. Using two real-world applications, we demonstrate the efficiency of our methods. The significance of our problem formulation and initial solutions is that it will serve as the foundation in assuring humans about machines making complex decisions.

The paper addresses the question: how can we be sure that an ML system behaves as intended. A straight forward method for this assessment is presented, some fundamental properties of the solution are discussed, and the method is tested exemplarily on two real word tasks.  Addressing the reliability of ML solutions is, in my opinion, a very important topic. I believe that the discussion will gain intensity. This contribution provides one systematic way to address one of the questions, concerning reliability. As the authors claim---the paper presents "an early attempt towards this goal.".   The paper is well written and organized. Still there seem to be a few textual errors, like page 2 line 45-46 "shows that it common to", page 4, line 188 "we use Hedge algorithm", and page 7, line "whose % compositions".  Starting on page 7, line 252 there are blanks before the % sign. The general rule is that no blank is used between a number and the percent sign, as is correctly written on page 8, line 294.  In addition I assume, that for the final version, the missing capitalization in the literature will be fixed, as in ". hedge", or "gaussian", "bayesian".  On page 1, line 13 is written "building trust between humans and machines". This sounds strange to me. I think it is all about the trust, that humans can have in machines, and I do not think that the trust a machine might or might not have in humans is addressed here. Same for page 3, line 95.  There is another question I'd like to ask. I'm not sure if the question or the answer to it is fitting in the text, but I ask anyway: If the process of algorithmic assurance makes more effort and needs more expensive labeling by the expert as the initial training of the ML solution, is this something that cannot be avoided, something that we need to accept if we want to get this assurance? And what about the data gather by the process of algorithmic assurance? We could use this data to improve the initial ML solution. Do we not consider this, because this line of thought will only lead us in a recursion. It might be just like not using a validation set for training. It might also be possible, that one could asses the quality of the initial ML solution and then use the new data to train a second, improved ML solution, and then, without assessing the new ML solution, assume that the error is no bigger than the error of the initial ML solution.   Young topic, many questions.  Post Rebuttal =========== The Author Feedback has sufficiently clarified my questions.


A convex program for bilinear inversion of sparse vectors
We consider the bilinear inverse problem of recovering two vectors, x 2 R L and w 2 R L , from their entrywise product. We consider the case where x and w have known signs and are sparse with respect to known dictionaries of size K and N , respectively. Here, K and N may be larger than, smaller than, or equal to L. We introduce`1-BranchHull, which is a convex program posed in the natural parameter space and does not require an approximate solution or initialization in order to be stated or solved. We study the case where x and w are S 1 -and S 2 -sparse with respect to a random dictionary, with the sparse vectors satisfying an effective sparsity condition, and present a recovery guarantee that depends on the number of measurements as L ⌦(S 1 + S 2 ) log 2 (K + N ). Numerical experiments verify that the scaling constant in the theorem is not too large. One application of this problem is the sweep distortion removal task in dielectric imaging, where one of the signals is a nonnegative reflectivity, and the other signal lives in a known subspace, for example that given by dominant wavelet coefficients. We also introduce a variants of`1-BranchHull for the purposes of tolerating noise and outliers, and for the purpose of recovering piecewise constant signals. We provide an ADMM implementation of these variants and show they can extract piecewise constant behavior from real images.

This paper concerns the problem of recovering two unknown signals from their entrywise product, assuming their signs are known. To constrain the problem further, the two signals are assumed to be sparse with respect to known dictionaries. The entrywise products naturally lead to hyperbola-shaped constraints. The known signs cut the space and help to restrict the feasible set to one branch of hyperbola for each product. This enables convexification of the constraint. Based on this, this paper proposes a convex formulation Eq (2) for the problem. The paper further proposes robust and total-variation extensions of the canonical formulation, and employs the ADMM to solve the resulting general formulation. On the theory side, this paper proves that (Theorem 1) when the dimension of the two vectors is slightly higher in order than the total nonzeros of the hidden sparse vectors (intuitively, the intrinsic degrees of freedom), the proposed convex formulation recovers the two unknown signals, provided the known dictionaries are iid Gaussian.   The problem considered in this paper seems new, although there is a similar prior work   -- Aghasi, Alireza, Ali Ahmed, and Paul Hand. "BranchHull: Convex bilinear inversion from the entrywise product of signals with known signs." arXiv preprint arXiv:1702.04342 (2017).   There are minor differences between the two:   1) For data model, the prior work does not assume sparsity. This causes (simple) changes of the objectives in the convex relaxations, although the constraints are exactly the same.   2) For recovery guarantee, both guarantees are near optimal and requires dimension proportional to the intrinsic degrees of freedom of the signals. The result presented in the current paper implies the prior one.   3) The proof strategies are very similar and there are differences in the technicalities though.   So while the results presented are novel, it is also incremental in view of the above points. I think it is worthwhile to clarify the technical novelty in the proof and highlight what significant changes are needed to obtain the current result.   Another concern is how relevant is the iid Gaussian assumption on the known dictionaries to practice. Thinking of application such as real-world phase retrieval, one of the dictionaries might be the partial Fourier matrix. Sec 3.2 includes a verification close to this practical situation, although the basis is random partial DCT matrix. I think more practical evaluation is needed.      Additional comments:   * Line 35: "Unlike prior studies like Ahmed et al. [2014], where the signals are assumed to live in known subspaces, ..." Ahmed et al. [2014] considers the blind convolution problem and so is considerably more different than just without the sparsity constraint. Is the target reference actually Aghasi et al 2016?    * Line 39: "Recent work on sparse BIP, specifically sparse blind deconvolution, in Lee et al. [2017] provides ..." looks confusing. Lee et al [2017] considers recovery of sparse rank-1 matrix, and at least superficially it is not about the BIP problem considered here or sparse blind deconvolution. Can the connection be stated more explicitly? Then it would start to make sense to discuss the deficiency of the previous work and hence highlight any improvement the current work can offer. Unless this can be clarified and direct comparison makes sense, I think the claim about the contribution in the next paragraph should be revised to avoid confusion.   * The bib information of Aghasi et al 2016 and Lee et al. [2017] seems problematic: the arXiv numbers should probably be switched.    * Connection to one-bit compressed sensing: as the signs of both unknown signals are known, and the known dictionaries are assumed to be "generic", i.e., Gaussian, there is a superficial connection to one-bit compressed sensing. Please comment on the connection, and discuss what happens if one approaches the problem as two one-bit compressed sensing problems.    * Proof of Lemma 1: Why one needs to multiply |y_\ell| to both sides of the inequalities above Line 340? This does not seem necessary.


Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?
This paper seeks to answer the question: as the (near-) orthogonality of weights is found to be a favorable property for training deep convolutional neural networks, how can we enforce it in more effective and easy-to-use ways? We develop novel orthogonality regularizations on training deep CNNs, utilizing various advanced analytical tools such as mutual coherence and restricted isometry property. These plug-and-play regularizations can be conveniently incorporated into training almost any CNN without extra hassle. We then benchmark their effects on state-of-the-art models: ResNet, WideResNet, and ResNeXt, on several most popular computer vision datasets: CIFAR-10, CIFAR-100, SVHN and ImageNet. We observe consistent performance gains after applying those proposed regularizations, in terms of both the final accuracies achieved, and faster and more stable convergences. We have made our codes and pre-trained models publicly available.
1 .

SUMMARY This work investigates various possible soft orthogonality constraints, promoting feature diversity in convolution layers, and yielding some improvement in image classification on CIFAR-10 and CIFAR-100.  STRENGTHS - Tested on popular classification models. - Pretty good overview of prior work. - SRIP is framed in an interesting way in equation 6, for use as a regularizer. - SRIP may be fairly cheap, computationally, when roughly approximated as proposed.  CRITICISMS - Because the proposed constraints enforce filters in convolution layers to be orthogonal to each other, they do not enforce the convolutional operators to be orthogonal; rather, they promote feature diversity in convolution layers. This is indeed proposed in the referenced "all you need is a good init" (10) paper as a regularizer. Such a constraint does not target the condition number of layer or address vanishing or exploding signals through a layer. This must be clearly stated and discussed. - Regarding 'scheme change': is it useful to even have a weight decay term for non-convolutional layers? Weight decay in DNNs is in practice enforced as an L2 norm on the weights which is essentially a penalty on the spectral radius (it bounds the largest singular value). Results without scheme change are not shown. The utility of 'scheme change' is stated but not proven. - In section 3.1: if W is m*n, then W'W is n*n, not m*m. Furthermore, it is stated that m ≤ n is undercomplete and necessary for the |W'W-I| constraint to work but this is inaccurate. This is actually correct elsewhere in the text but in this section, some symbols appear incorrectly arranged. If m < n, then W may be overcomplete, restricting W'W is problematic, and WW' might work. If m > n, then it may be undercomplete and restricting W'W works. Finally, if m = n, then W may be complete (and restricting W'W works).     -- "if and only if W is undercomplete (m ≤ n)" -> '(n ≤ m)'     -- "For overcomplete W (m > n), its gram matrix ∈ m × m cannot be even close to identity, because its rank is at most n" -> '(n > m)' and 'rank is at most m' - Final performace values tend to be very close together. What is the variance when rerunning experiments? Rerunning an exact reproduction of wide resnet, variance contains the results of multiple papers that claimed SOTA over wide resnets.     -- The authors state that MC outperforms the baseline with a 6.97% error vs 7.04% error on CIFAR 10 and 25.42% for both on CIFAR-100. The performance appears identical for both. - There is no description of the way validation data was split from training data. Was there always a validation set separate from the test set? This should be detailed and made clear. If testing was done on the test set (as in wide resnet), the evaluation is incorrect. - It would be useful to compare against a norm constraint on neighbouring activations ("Regularizing RNNs by Stabilizing Activations" - 2015) or a norm constraint on the gradients between neighbouring layers ("On the difficulty of training recurrent neural networks." - 2013).  MINOR COMMENTS - Orthogonality constraints may be more useful for RNNs and GAN discriminators than for ResNets, which already have a stable gradient guarantee, or for encouraging feature diversity in convolution lahyers. - "which provides a potential link between regularizing orthogonality and spectrum." -- it's the same thing - "we gradually reduce λ (initially 0.1) by factors of 1e-3, 1e-4 and 1e-6" - not 'by factors of', rather 'to'     - learning rate "decreased by a factor of 0.2" - 'by a factor of' means divided by, which would in this case mean the lr is increasing     - similarly, 'by a factor of 0.1' - It's surprising that "DSO always outperforms selective regularization". - SRIP is interesting. There should be a discussion on the time cost of different regularization approaches. With such a rough approximation as proposed (2 power method iterations), SRIP reduces computational cost from O(n**3) for soft orthogonality to O(n**2); from matrix-matrix multiplication to matrix-vector multiplication. - Perhaps there is a better lambda schedule for soft orthogonality that would make it outperform SRIP. - The MC trick may need a higher weight at first than soft orthogonality and SRIP due to a sparser gradient signal. - If (4) is an approximation, it's not "equivalent".  TYPOS     - in Assumption 1: z in [R]^n - "but unnecessarily appropriate" - not necessarily - "phase(direction)" - space - "without bothering any assistance components." - strange wording - "could possibly to mutually" - 'to' -> 'be' - "structure is most benefit at the initial stage" - 'beneficial' - eq 5 missing vertical bar  OVERALL OPINION The evaluation of which orthogonality regularizers are possible and useful, and by how much, is a useful one for the community. The SRIP formulation is interesting. However, the scope of the analysis should be expanded and the robustness of comparisons should be improved with statistical analysis. The effect of the proposed regularizations on learned representations and on network dynamics should be clearly identified and discussed. Some errors should be corrected. I would encourage submitting this work to a workshop.  UPDATE Significant work was done in the rebuttal to address the criticisms.  Although the authors did not address the significance of results in Table 1, they (a) confirmed the reliability of results in Table 2 and (b) performed additional experiments on both SVHN and ImageNet that go a long way toward convincing me that SRIP could be a reliably useful regularizer. The authors also included additional experiments with constraints on gradient norm change, further demonstrating superior results with SRIP.  The authors demonstrated a will to correct errors and clarify misleading parts of the text. I would especially encourage that they follow through with clarifying in the beginning that the proposed work enforces orthogonality across filters, encouraging filter diversity, and does not enforce orthogonality "on linear transformations between hidden layers" (second paragraph) in the way that some of the references do. The authors clarified some concerns such as whether hyperparameters were tuned on the validation data.  Authors responded qualitatively about time complexity -- this should be in big O notation and would work in their favor to advertise in the text, along with the qualitative note about perceived wall clock time, as the low cost of the method is attractive.  In light of the additional results and the clarification going into the revision, I will raise my rating from a 4 to a 7.


Multi-objective Maximization of Monotone Submodular Functions with Cardinality Constraint
We consider the problem of multi-objective maximization of monotone submodular functions subject to cardinality constraint, often formulated as max |A|=k min i∈{1,...,m} f i (A). While it is widely known that greedy methods work well for a single objective, the problem becomes much harder with multiple objectives. In fact,  showed that when the number of objectives m grows as the cardinality k i.e., m = Ω(k), the problem is inapproximable (unless P = N P ). On the other hand, when m is constant Chekuri et al. (2010) showed a randomized (1 − 1/e) − approximation with runtime (number of queries to function oracle) n m/ 3 . We focus on finding a fast and practical algorithm that has (asymptotic) approximation guarantees even when m is super constant. We first modify the algorithm of Chekuri et al. (2010) to achieve a (1 − 1/e) − approximation for m = o( k log 3 k ), with → 0 as k → ∞. This demonstrates a steep transition from constant factor approximability to inapproximability around m = Ω(k). Then using Multiplicative-Weight-Updates (MWU), we find a much fasterÕ(n/δ 3 ) time asymptotic (1 − 1/e) 2 − δ approximation. While the above results are all randomized, we also give a simple deterministic (1 − 1/e) − approximation with runtime kn m/ 4 . Finally, we run synthetic experiments using Kronecker graphs and find that our MWU inspired heuristic outperforms existing heuristics.

The paper studies a robust formulation of maximizing a submodular function subject to a cardinality constraint. Specifically, the goal is to maximize the minimum of m submodular functions f_1,..., f_m subject to picking at most k elements. When there is no restriction on k and m, the problem can be hard to approximate. The paper focuses on the regime where m << k and in this regime, it is possible to avoid the hardness and obtain approximation solution.  The algorithm is inspired by previous work by CCPV'07. Previously the algorithm needs to guess the elements with large value because rounding only works when all elements have small values. Furthermore, previous work focused on getting polynomial time and uses a slow continuous greedy approach. To address the first problem, the paper focuses on a special case m<
Reviewer 3
The paper proposes three efficient multiobjective submodular maximization algorithms.  I like this paper, and has no clear objection to the paper. The techniques (using one-pass procedure for picking the heavy elements, separating MWU and the continuous greedy) are simple and effective, and seem to be applied for more general problems.    [minor] - l.21: The first paragraph (many well known ...) is arguable. There are many non-submodular combinatorial optimization problems.  - l.60: the approximation ratio of multilinear extension-based method must be (1 - 1/e) - \epsilon. - l.89: Please write the complexity of (1 - 1/e) approx algorithm explicitly. I think it already gives significant improvement from n^{O(1/eps)} to O(poly(n, 1/eps)), and the actual bottleneck of Algorithm 1 is this factor (usually it exceeds n^{100}). - l.129: it wil be better to write x = (x_1, ..., x_n) instead of x = {x_1, ...,x_n}.  [very minor] - l.129: I guess you used $$ ... $$ for the display math. As in https://nips.cc/Conferences/2018/PaperInformation/StyleFiles it should be avoided because it breaks line numbering.


Global Non-convex Optimization with Discretized Diffusions
An Euler discretization of the Langevin diffusion is known to converge to the global minimizers of certain convex and non-convex optimization problems. We show that this property holds for any suitably smooth diffusion and that different diffusions are suitable for optimizing different classes of convex and non-convex functions. This allows us to design diffusions suitable for globally optimizing convex and non-convex functions not covered by the existing Langevin theory. Our non-asymptotic analysis delivers computable optimization and integration error bounds based on easily accessed properties of the objective and chosen diffusion. Central to our approach are new explicit Stein factor bounds on the solutions of Poisson equations. We complement these results with improved optimization guarantees for targets other than the standard Gibbs measure.

This paper studies nonconvex optimization problem using Euler discretization on Ito diffusions, and derives the convergence rate of the investigated discretized diffusions. Specifically, the authors apply Poisson equation to analyze the convergence of discretized diffusions in terms of expected function value and prove a convergence rate of O(1/\epsilon^2) under several seemingly stringent assumptions. In addition, the authors also give some examples that are not covered by existing Langevin theory but can be illustrated by the theory in this paper. I notice that the authors mentioned one recent work in the literature, i.e., [26], which is highly related to this submission. However, I didn’t find a careful comparison between this submission and [26] in the related work or the main theory section (Section 3). Since the O(1/epsilon^2) convergence results seems worse than the O(1/\epsilon) rate proved in [26],yet  this work makes more assumptions beyond the smooth and dissipativity assumptions, it is very important to discuss the connection and difference between this submission and [26] thoroughly.      The strengths of this paper are given as follows: 1. This paper studies a Euler discretization on general Ito diffusions, which is beyond the framework of conventional Langevin dynamics. 2. This paper presents some examples to showcase the applications of the main theory.   The weakness of this paper: 1. In [22], the convergence rate of SGLD contains a spectral gap which is exponential in dimension d and inverse temperature \gamma. Similarly, the result for GLD (the full gradient version) in [26] has also such a dependence on d and \gamma. Furthermore, there is a lower bound for the metastable exiting time of SDE for nonconvex functions in Bovier et al., (2004), which has exponential dependence on d and \gamma too. However, this paper seems to hide many factors including the spectral gap in the main theory. The authors should explicitly indicate these dependences and make a fair and thorough comparison with previous work. 2. The results for nonconvex optimization is claimed to be \mathcal{O}(1/\epsilon^2), while the complexity in [26] is \tilde(O)(1/\epsilon) for the same algorithm (GLD). The authors should explain why their result looks inferior to the existing work.  3. In Section 5, the authors just simply combine the argument in Theorem 3.1 and Proposition 3.4 in [22]. Some important discussions are missing, such as the optimal choice of the inverse temperature parameter \gamma, and how it affects the convergence rate. I believe the choice of \gamma will affect both the ergodicity and discretization results. Besides, the authors attempt to improve the expected suboptimality using non-Gibbs distributions. However, a simple quadratic example is not convincing enough, and the impacts of using non-Gibbs distributions on the ergodicity were not well explained. 4. The convergence rate highly depends on the parameter n, I would like to see some discussions on this parameter (use some examples or provide an upper bound).  5. In condition 3, the authors mention \tilde \mu_n(u_f) = \zeta_1. I noticed that the notation \tilde \mu_n(u_f) has appeared before in (2.6), are they the same? 6. A typo in Line 72: it is the inverse of spectral gap, i.e., \lambda^{-1} that is exponential in d and \gamma.  7. The organization can be improved, the authors are better to clearly state the assumptions on objective function, rather than write them with plain text.   Reference:  Bovier, A., Eckhoff, M., Gayrard, V. and Klein, M. (2004). Metastability in reversible diffusion processes i: Sharp asymptotics for capacities and exit times. Journal of the European Mathematical Society 6 399–424.  ===== After I read the authors' response, my concern on the comparison with [26] was addressed. Please incorporate the comparison with [26] in your response into the camera ready.


Depth-Limited Solving for Imperfect-Information Games
A fundamental challenge in imperfect-information games is that states do not have well-defined values. As a result, depth-limited search algorithms used in singleagent settings and perfect-information games do not apply. This paper introduces a principled way to conduct depth-limited solving in imperfect-information games by allowing the opponent to choose among a number of strategies for the remainder of the game at the depth limit. Each one of these strategies results in a different set of values for leaf nodes. This forces an agent to be robust to the different strategies an opponent may employ. We demonstrate the effectiveness of this approach by building a master-level heads-up no-limit Texas hold'em poker AI that defeats two prior top agents using only a 4-core CPU and 16 GB of memory. Developing such a powerful agent would have previously required a supercomputer.

Summary:  The paper proposes a modification of the continual resolving algorithm for depth-limited search in imperfect information games. Instead of counterfactual value network mapping joint beliefs to counterfactual values, it suggests using a function that maps a public state to a set of values of diverse best response strategies again an approximate Nash equilibrium strategy. This function can be precomputed, or realized either by function approximation of by Monte Carlo simulation based on pre-computed strategies. The authors show that this approach performs well in poker with small computational requirements.   Quality:  The core ideas of the paper are correct and interesting. There are several technical details that are unclear or overstated.  1) The most important problem are the inconsistencies regarding the blueprint strategy. The paper assumes at several places that the player plays based on a precomputed strategy \sigma^*_1. At the same time, the paper says that the player never plays based on the blueprint strategy. Since the blueprint strategy is created using abstraction, which leads to huge exploitability in HUNL, it is likely substantially different from the strategy computed and played online, especially when the opponent uses actions not included in the abstraction. All the theoretical guarantees relay on the player playing based on the blueprint strategy. Therefore, they make only a very limited sense and serve only as an inspiration for the heuristic approach actually evaluated in the paper. This is not clearly stated. I do not see how the theory can be useful unless the blueprint strategy already solves the game and therefore, there is no need for online computation. I ask the authors to elaborate on this in the rebuttal.  2) The second important weakness of the paper is the lack of experimental comparison with the state of the art. The paper spends whole page explaining reasons why the presented approach might perform better under some circumstances, but there is no hard evidence at all.  What is the reason not to perform an empirical comparison to the joint belief state approach and show the real impact of the claimed advantages and disadvantages? Since this is the main point of the paper, it should be clear when the new modification is useful.  3) Furthermore, there is an incorrect statement about the performance of the state of the art method. The paper claims that "The evidence suggests that in the domain we tested on, using multi-valued states leads to better performance." because the alternative approach "was never shown to defeat prior top AIs". This is simply incorrect. Lack of an experiment is not evidence for superiority of the method that performed the experiment without any comparison.  4) The rock-paper-scissors example is clearly inspired by an example that appeared in many previous work. Please, cite the source appropriately.  5) As explained in 1), the presented method is quite heuristic. The algorithm does not actually play the blueprint strategy, only few values are used in the leaf states, which cannot cover the whole variety of the best response values. In order to assess whether the presented approach might be applicable also for other games, it would be very useful to evaluate it on some substantially different domains, besides poker.  Clarity:  The paper is well written and organized, and it is reasonably easy to understand. The impact of the key differences between the theoretic inspiration and the practical implementation should be explained more clearly.  Originality:  The presented method is a novel modification of continual resolving. The paper clearly explains the main distinction form the existing method.   Significance:  The presented method seems to substantially reduce the computational requirements of creating a strong poker bot. If this proofs to be the case also for some other imperfect information games, it would be a very significant advancement in creating algorithms for playing these games.   Detailed comments:  190: I guess the index should be 1 339: I would not say MCCFR is currently the preferred solution method, since CFR+ does not work well with sampling 349: There is no evidence the presented method would work better in stratego. It would depend on the specific representation and how well would the NN generalize over the types of heuristics.  Reaction to rebuttal:  1) The formulation of the formal statement should be clearer. Still, while you are using the BR values from the blueprint strategy in the computation, I do not see how the theory can give you any real bounds the way you use the algorithm. One way to get more realistic bounds would be to analyze the function approximation version and use error estimates from cross-valiadation. 2) I do not believe head-to-head evaluation makes too much sense because of well known intransitivity effects. However, since the key difference between your algorithm and DeepStack is the form of the used leaf evaluation function, it would certainly not take man-years to replace the evaluation function with the joint belief in your framework. It would be very interesting to see comparison of exploitability and other trade-offs on smaller games, where we can still compute it. 4) I meant the use of the example for save resolving. 5) There is no need for strong agents for some particular games to make rigorous evaluation of equilibrium solving algorithms. You can compute exploitability in sufficiently large games to evaluate how close your approach is to the equilibrium. Furthermore, there are many domain independent algorithms for approaximating equilibriua in these games you can compare to. Especially the small number of best response values necessary for the presented approach is something that would be very interesting to evaluate in other games.  Line 339: I just meant that I consider CFR+ to be "the preferred domain-independent method of solving imperfect-information games", but it is not really important, it was a detailed comment.


Online Structured Laplace Approximations for Overcoming Catastrophic Forgetting
We introduce the Kronecker factored online Laplace approximation for overcoming catastrophic forgetting in neural networks. The method is grounded in a Bayesian online learning framework, where we recursively approximate the posterior after every task with a Gaussian, leading to a quadratic penalty on changes to the weights. The Laplace approximation requires calculating the Hessian around a mode, which is typically intractable for modern architectures. In order to make our method scalable, we leverage recent block-diagonal Kronecker factored approximations to the curvature. Our algorithm achieves over 90% test accuracy across a sequence of 50 instantiations of the permuted MNIST dataset, substantially outperforming related methods for overcoming catastrophic forgetting.

*** Edit after the rebuttal period: I have read other reviews and the author feedback. The experiments in this paper are solid and the storyline is clear, however, the proposed method is of arguably low novelty. I will keep my evaluation and rating as is. *** Summary: The paper considers online structured Laplace approximation for training neural networks to mitigate catastrophic forgetting. The idea is to perform Laplace approximation for a task to obtain a Gaussian approximation to the posterior, use this Gaussian as the prior for the weights for next task, and repeat. This results in an addition of an L2 regularisation term to the objective and requires the Hessian of the objective evaluated at the MAP estimate — this is approximated by a Kronecker factored empirical Fisher information matrix. Adding multiple penalties as in EWC and reweighting the empirical FIM term were considered. The take-home message seems to be that using a non-diagonal approximation to the posterior gives a significant gain for all methods across many supervised training tasks.  Comments:  The paper is very well written and in my opinion deserves acceptance. The key idea is relatively simple in hindsight, that is to use structured curvature information for the Laplace approximation (Ritter et al, 2018) in the continual learning setting. Previous works such as EWC, which can be interpreted as online Laplace approximation, only consider a diagonal approximation to the Fisher information matrix. The experiments showed that the addition of the structured approximation is clearly beneficial to online learning, in a similar fashion to the same structured approximation yielding better Laplace approximation in the batch setting (Ritter et al, 2018).  This paper combines several existing techniques and approximations to form a principled approach to continual learning. One could argue that the novelty of this paper is incremental, but the practical advantage of the proposed method seems significant and, as such, could generate interests in the community as continual learning has been an active area of research in the last couple of years.  One thing I dislike is that this method relies on a validation set to tune the regularisation hyper-parameters and the performance is very sensitive to these hyperparams, and it feels unsatisfying that vanilla (approximate) online Bayesian updating is not sufficient to warrant good performance. The variational continual learning framework of Nguyen et al (2018) for example does not introduce any additional hyperparameters to reweight the regularisation term and the expected likelihood term, while still performing quite well with a diagonal Gaussian approximation.


Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimization
As application demands for zeroth-order (gradient-free) optimization accelerate, the need for variance reduced and faster converging approaches is also intensifying. This paper addresses these challenges by presenting: a) a comprehensive theoretical analysis of variance reduced zeroth-order (ZO) optimization, b) a novel variance reduced ZO algorithm, called ZO-SVRG, and c) an experimental evaluation of our approach in the context of two compelling applications, black-box chemical material classification and generation of adversarial examples from black-box deep neural network models. Our theoretical analysis uncovers an essential difficulty in the analysis of ZO-SVRG: the unbiased assumption on gradient estimates no longer holds. We prove that compared to its first-order counterpart, ZO-SVRG with a two-point random gradient estimator could suffer an additional error of order O(1/b), where b is the mini-batch size. To mitigate this error, we propose two accelerated versions of ZO-SVRG utilizing variance reduced gradient estimators, which achieve the best rate known for ZO stochastic optimization (in terms of iterations). Our extensive experimental results show that our approaches outperform other state-of-the-art ZO algorithms, and strike a balance between the convergence rate and the function query complexity.

This paper extensively investigates the advantage of variance reduction for stochastic zeroth-order in non-convex optimization (where one does not have access to the gradient of the objective). They consider three different approaches to zo gradient estimation, including  RandGradEst, Avg-RandGradEst, and CoordGradEst. For each method, they propose a convergence guarantee. The main contribution (expressed in Theorem 3) is the derivation of the improved rate O(1/T) for CoordGradEst using variance reduction. The result is novel and can potentially have a high impact on non-convex optimization.   Although the authors have done a good work to provide a rich theoretical result, I think that the presentation of the paper can be improved a lot. Here, I’ve listed my concerns regarding results and the presentation:   1. My first concern is the dependency of all convergence result to the number of samples n. We can not hide this factor in the O(.) notation because it can crucially change the total time complexity of algorithms (see the seminal paper “Stochastic Variance Reduction for Nonconvex Optimization” where the number of samples is clearly presented in time complexity). 2. The paper uses existing proof techniques of stochastic non-convex optimization and variance reduction without citing them. For example, the Lyapunov function of Eq. 104 is previously used in “Stochastic Variance Reduction for Nonconvex Optimization”. 3. Although the authors have tried to highlight the difference between the difference between different 0-order gradient estimates, I am not convinced that CoordGradEst is the best approach with such a significant improvement in the convergence. The empirical results (in the experiments section) also show that Avg-RandGradEst outperforms as well as CoordGradEst. I think that the source of the convergence gap is the loose bound of Eq 37. If one can prove that \had{\nabla} f_i is Lipschitz in expectation then a same converge rate O(1/T) can be driven for  Avg-RandGradEst and even RandGradEst.  4.  The proof technique behind all three theorems is the same. I highly recommend to integrate the similar parts of proof in lemmas and propositions instead of repeating all the derivations.  If authors represent the upper-bound of proposition 1 for all three 0-order gradient estimates, then reads can understand better the difference between these three approaches. I think that the last term of upper bound vanishes for CoordGradEst and this the source of the improved rate (still I think that this bound can be improved for RandGradEst). Regarding the bound of Eq 99, I could not reach the same upper bound using Eqs (34)-(36).  5. I recommend to first analyze the effect of variance reduction on the smoothed function f_\mu instead of the original function f. Since 0-order gradient estimate provides an unbiased estimate on f_\mu, this helps to simplify the analysis. Then we can relate f_\mu to f in terms of suboptimality. Furthermore, the objective f_\mu as a smooth version of the main function might be more favorable for non-convex optimization.  --------------------------------------------- I have read the author response. They addressed most of my concerns, hence I've updated my score. Regarding the 3rd comment, If they assume that stochastic gradients are Lipschitz, they might be able to achieve better bound in Eq. 37.


Learning without the Phase: Regularized PhaseMax Achieves Optimal Sample Complexity
The problem of estimating an unknown signal, x 0 ∈ R n , from a vector y ∈ R m consisting of m magnitude-only measurements of the form y i = |a i x 0 |, where a i 's are the rows of a known measurement matrix A, is a classical problem known as phase retrieval. This problem arises when measuring the phase is costly or altogether infeasible. In many applications in machine learning, signal processing, statistics, etc., the underlying signal has certain structure (sparse, low-rank, finite alphabet, etc.), opening of up the possibility of recovering x 0 from a number of measurements smaller than the ambient dimension, i.e., m < n. Ideally, one would like to recover the signal from a number of phaseless measurements that is on the order of the "degrees of freedom" of the structured x 0 . To this end, inspired by the PhaseMax algorithm, we formulate a convex optimization problem, where the objective function relies on an initial estimate of the true signal and also includes an additive regularization term to encourage structure. The new formulation is referred to as regularized PhaseMax. We analyze the performance of regularized PhaseMax to find the minimum number of phaseless measurements required for perfect signal recovery. The results are asymptotic and are in terms of the geometrical properties (such as the Gaussian width) of certain convex cones. When the measurement matrix has i.i.d. Gaussian entries, we show that our proposed method is indeed order-wise optimal, allowing perfect recovery from a number of phaseless measurements that is only a constant factor away from the optimal number of measurements required when phase information is available. We explicitly compute this constant factor, in terms of the quality of the initial estimate, by deriving the exact phase transition. The theory well matches empirical results from numerical simulations.

This paper presents a sharp asymptotic analysis of the regularized phase-max estimator for structured phase retrieval (PR) problems. The phase-max algorithm is a recently proposed Linear Programming based estimator for PR. The finite sample as well as the sharp asymptotic performance of phase-max for unstructured phase retrieval was well understood from prior work. The authors propose a natural modification  by penalizing the usual phase-max objective with a regularizer $f$. For structured PR, prior work showed that for a $k$-sparse signal vector, L1-penalized phase-max recovers the signal with $m = O(klog(n/k))$ samples given a good enough initialization. The primary contribution of this work is (i) extending the result to absolutely scalable regularizers which allows for other kinds of structural assumptions on the signal (ii) complementing the finite sample with a high dimensional asymptotic result giving an explicit formula for the asymptotic sampling  ratio $\alpha = m/n$ required to achieve exact recovery in terms of the quality of the initialization and the oversampling ratio $\delta = d/m$ where $d$ is the effective degrees of freedom of the signal vector. The main tool used to establish this result is the Convex Gaussian Min-Max (CGMT) framework which is a well-established technique from prior work.  The quality and clarity of the work can be improved. Several of the statements of the theorems and lemmas are not very precise and some of the arguments are non-rigorous. However, the results are definitely interesting and plausible (see detailed comments below). They are also supported by simulations. Section 4.5 is clearly incomplete and ends abruptly. Citations in paragraph 509-515 are informal notes. To the best of my knowledge, the formula for the phase transition presented by the authors was not known before and is a significant and original contribution. While CGMT is a well-established technique, applying it to this problem seems to require some original and interesting problem specific analysis.  Detailed Comments about proofs: Lines 409-412: The CGMT lemma requires additional assumptions which are omitted and not verified in the proof. The compactness assumption on $S_w$ and $S_u$ is stated but not verified when invoking this lemma in 545-547. This usually requires some technical arguments. Lemma 3, Eq 23: The claimed $n \rightarrow \infty$ limit depends on $n$.  Lines 434-436: Approximation of $f$ by the linear underestimator is justified non-rigorously. Likewise in lines 471-475.  Lines 480-482: Can the authors state which lemma/theorem of [Thrampoulidis et al, 2016] they are invoking to justify changing the order of min-max? I am unable to find such a result in that paper and verify their claim.  Pages 17-18: To apply the CGMT theorem, the authors need to verify that the AO optimal value and the norm of the AO optimal solution converge in probability. It appears to me this has not been done rigorously. In particular the AO and PO have a distinctive feature that they depend on additional randomness $q$ apart from the usual randomness $A, g, h$. The authors don’t state how they handle this. CGMT is a general purpose theorem and most of the work involved is in verifying its technical conditions and it would nice to have these details in the appendix.  Lines 198-201: I don’t think the citation [9] calculates the statistical dimension of the descent cone of this objective. I can see this was being done in the appendix section 4.5 which is incomplete.  Other minor comments: Lines 98-100: It should be clarified that the paper by Soltanolkotabi [33] breaks the \emph{local} sample complexity barrier (sample complexity required for recovery when initialized closed enough to the truth) and not the global complexity barrier.  Lines 133-138: The reference stated does not justify the optimality of the sample complexity. To my understanding, these references give the sample complexity required for certain estimators to succeed for linear measurements. It might be more appropriate to have a reference for an information theoretic lower bound/ identifiability lower bound in the specific context of phase retrieval since the title of the paper also claims optimality.  Line 213: orthogonal projection of sub-diff .. into the orthogonal… : Remove first orthogonal.  Line 216: Missing linebreak at Assumption 1.  Figure 1(b): missing legend. Lines 237-241: Use \begin{remark} Line 187: Missing linebreak at Example 1.  Lines 167-168: Has the claim about the function $R$ been proved anywhere in the paper? If it is a known result, it needs a citation.  Lines 169-171: Is the requirement on $R(\delta)$ for all $n$?  --------------------------------------------------------------------  POST-REBUTTAL:  My primary concern  regarding this paper was that some of statements and arguments appeared non-rigorous and imprecise to me. The authors addressed these issues in their rebuttal. Furthermore since I do think this result is interesting to the NIPS community, I am increasing my score to 7.


Structured Local Minima in Sparse Blind Deconvolution
Blind deconvolution is a ubiquitous problem of recovering two unknown signals from their convolution. Unfortunately, this is an ill-posed problem in general. This paper focuses on the short and sparse blind deconvolution problem, where the one unknown signal is short and the other one is sparsely and randomly supported. This variant captures the structure of the unknown signals in several important applications. We assume the short signal to have unit 2 norm and cast the blind deconvolution problem as a nonconvex optimization problem over the sphere. We demonstrate that (i) in a certain region of the sphere, every local optimum is close to some shift truncation of the ground truth, and (ii) for a generic short signal of length k, when the sparsity of activation signal θ k −2/3 and number of measurements m poly (k), a simple initialization method together with a descent algorithm which escapes strict saddle points recovers a near shift truncation of the ground truth kernel.

This paper considers sparse blind deconvolution. This is an inverse problem where we observe the (cyclic) convolution of two unknown signals: one sparse, and one short. The aim is to recover both signals, up to fundamental indeterminacies. This is done through non-convex optimization with a soft sparsity promoting cost function. The main contributions are an analysis of regimes where the non-convexity is benign. Importantly, there is no noise in the proposed model.  This is an interesting problem. Both the problem and the proposed approach I would deem fairly original. Exposition is satisfactory, and results are interesting.  There are two main points I would like the authors to clarify. A priori, I expect the authors will be able to resolve these two doubts I have, which is why my initial rating is high. But if I cannot be convinced, I will have to lower my grade.  First, does the proposed theory cover more than a subclass of this problem which is trivial? Specifically, see the rate at L69. What does this mean exactly? From the model detailed after L69, about m\theta entries of x_0 are nonzero. If k is such that there is a good probability that just one nonzero entry is separated from any other nonzero entry by at least k zeros, then the kernel a_0 can be read off directly from the observation just by spotting the presence of two exact zeros around the signal, which is then only scaled by the corresponding g_i. Thus, to make sense of the sparsity rate (and to know if it is "relatively dense" as stated in the paper), it should be shown that under the proposed setting, this situation does not occur with large probability. Otherwise, the problem appears to be trivial. This issue could also be resolved by adding noise to the observation y, in which case the exact zeros would not appear, and it wouldn't be all that easy to locate the separated occurrences. More generally, it would be nice to treat the presence of noise, but this is not a requirement.   Second, line 65, the word "benign" (and many other places where this sentiment is echoed): it is unclear to me that this effect is benign. Is it not the case that the shift truncation could eliminate most if not all of the signal? If that is the case, what is it really that we can hope to recover? In other words, does the measure proposed on line 253 really capture closeness to a meaningful version of a_0? Or are some of these shift truncated versions of a_0 somehow significantly shortened to the point that we didn't recover much?     Other comments:  Figure titles, legends, axis labels etc.: a lot of them are illegible. Please fix this.  Eq. (3): at this point, the modulo operator has not been defined yet. On that note: notation would be a lot nicer with 0 indexing, as this would ease all the modulo business.  Line 44: , -> .  L62,63: this is nice intuition.  Page 2, footnote 2: is it on purpose that y_1 is not moved? If so, please specify so readers are not left wondering if this is a typo.  L78: what is entry-wise \ell^p norm? Also, this could conceivably interfere with the definition of _2 for operator norm. I understand that this is why you use the Frobenius norm for vectors. An alternative is to write \|.\|_{op}. In any case: there is some confusion in the paper itself about these norms. See for example eq. (9) where the 2 norm is presumably a Frobenius norm, since the input is a vector.  Eq. (6): the matrix seems wrongly typeset  L108: This justification (footnote 8) is a bit weak: one needs to argue first that YY'/m concentrates around its mean, then to argue that the same goes for the inverse matrix square root of it, and this this all happens nicely considering YY' appears in a product with other random variables.  Footnote 5: you get m-1 such contributions -- then, how can I understand this as small?  L143 (and likely elsewhere): optima is the plural of optimum (and minima/minimum, maxima/maximum)  L148: small \kappa, but eq. (19) seems to say small kappa (condition number) is bad; what am I missing?  L204: semidefinite -> should be definite. L205: convex -> should be locally geodesically convex L209-211: I do not understand this statement. Please clarify. L220, end: that -> such that L249: what does big mean here? very sparse or not very sparse? L249: move footnote mark outside parenthesis (and after period) L253: recover+y   ####### Post rebuttal #######  I thank the reviewers for their thorough response to my two main concerns. Based on their convincing explanations, I maintain my assessment of 8. I strongly encourage the authors to include the points raised in their rebuttal in their paper in some form.


Online Learning of Quantum States
Suppose we have many copies of an unknown n-qubit state ⇢. We measure some copies of ⇢ using a known two-outcome measurement E 1 , then other copies using a measurement E 2 , and so on. At each stage t, we generate a current hypothesis ! t about the state ⇢, using the outcomes of the previous measurements. We show that it is possible to do this in a way that guarantees that |Tr(E i ! t ) Tr(E i ⇢)|, the error in our prediction for the next measurement, is at least " at most O n/" 2 times. Even in the "non-realizable" setting-where there could be arbitrary noise in the measurement outcomes-we show how to output hypothesis states that incur at most O( p T n ) excess loss over the best possible state on the first T measurements. These results generalize a 2007 theorem by Aaronson on the PAC-learnability of quantum states, to the online and regret-minimization settings. We give three different ways to prove our results-using convex optimization, quantum postselection, and sequential fat-shattering dimension-which have different advantages in terms of parameters and portability.

Summary:  This paper considers the problem of learning mixed quantum states, in the sense of being able to predict the probability that a two-outcome measurement E would accept on the unknown quantum state rho. Prior work has considered this problem in the statistical setting, where it is assumed that there is a distribution D over two-outcome measurements, we have sample access to the distribution D, and our goal is to estimate a mixed quantum state so that with high probability for a measurement E drawn from D, the acceptance probability of the estimated state is close to the acceptance probability of the unknown state. This work considers the same problem in the online setting, where it is no longer assumed that measurements are drawn iid from some unknown distribution D. Instead, an arbitrary sequence of measurements is chosen by a (possibly adaptive) adversary, and the goal is design learning algorithms with either mistake bounds (i.e., bounds on the number of times the predicted acceptance probability is wrong my more than epsilon), or regret bounds (i.e., a bound on the cumulative loss of the predicted acceptance probabilities). The online setting is especially interesting given that the sequence of measurements being learned from is usually constrained by nature and chosen by an experimentalist (rather than being iid).   The paper has two main results:  The first result shows that (in the realizable setting) there exists an algorithm that receives a sequence of two-outcome measurements E_1, E_2, ... one at a time and for each outputs an estimated state omega_t such that the acceptance probability of E_t on omega_t is within epsilon of the acceptance probability of E_t on the unknown hidden state rho for all but O(n/epsilon^2) rounds. This feedback given to the algorithm comes in the form of an estimate b_t of the true acceptance probability of E_t on rho, which should be accurate to within error epsilon/3.  The second result applies in the non-realizable setting, where we do not assume that the sequence of losses truly arise as the acceptance probabilities of some two-outcome measurements applied to some fixed mixed quantum state. Instead, we suppose that for each measurement E_t there is an L-Lipschitz convex loss function ell_t mapping acceptance probabilities to losses. The theorem guarantees that there exists an algorithm with regret bounded by O(L sqrt(Tn)) after T measurements, where n is the number of qubits.   Section 3 shows that both of the main results can be achieved using variants of the regularized follow the leader algorithm or the matrix multiplicative weights algorithm. The regret bounds for these algorithms follow the standard proofs, however some modifications are necessary to deal with the fact that the input domain is complex. The mistake bound is a corollary of the regret bounds.  Sections 4 and 5 provide alternative proofs for the two main results using different techniques. In particular, Section 4 provides an alternative mistake bound of O(n/epsilon^3 log(n/epsilon)) using repeated application of a post-selection operator (and this algorithm appears to be significantly different from the FTRL and multiplicative weights algorithms). In section 5, the authors show that a learning algorithm with regret O(L sqrt(nT) log^{3/2}(T)) exists by bounding the sequential fat shattering dimension of the prediction problem.  Comments and Questions:  I am relatively unfamiliar with quantum computing and the related literature, but the problem in this paper seems to be well motivated, especially the shift to the online setting from prior work using statistical assumptions on the source of measurements. The main results are clearly stated and convincing, especially given that the authors present several different techniques all arriving at similar bounds. For the broader NIPS community, I think it is interesting to see these methods applied to problems somewhat distant from more typical prediction problems. It might be helpful to include slightly more background information about the underlying problem and what the benefits of having online prediction algorithms might be.  In the abstract it is mentioned that we measure multiple copies of the unknown state using the same two-outcome measurement E_1. I assume that multiple measurements are used to estimate the acceptance probabilities, since each measurement leads only to the binary accept or reject outcome. In the rest of the paper we suppose we use each measurement only once, but as feedback we get an estimate of the acceptance probability. It might be worth mentioning this again in the main body somewhere.


PCA of high dimensional random walks with comparison to neural network training
One technique to visualize the training of neural networks is to perform PCA on the parameters over the course of training and to project to the subspace spanned by the first few PCA components. In this paper we compare this technique to the PCA of a high dimensional random walk. We compute the eigenvalues and eigenvectors of the covariance of the trajectory and prove that in the long trajectory and high dimensional limit most of the variance is in the first few PCA components, and that the projection of the trajectory onto any subspace spanned by PCA components is a Lissajous curve. We generalize these results to a random walk with momentum and to an Ornstein-Uhlenbeck processes (i.e., a random walk in a quadratic potential) and show that in high dimensions the walk is not mean reverting, but will instead be trapped at a fixed distance from the minimum. We finally analyze PCA projected training trajectories for: a linear model trained on CIFAR-10; a fully connected model trained on MNIST; and ResNet-50-v2 trained on Imagenet. In all cases, both the distribution of PCA eigenvalues and the projected trajectories resemble those of a random walk with drift.

The paper looks at the dynamics of training in DNNs through the lens of PCA in comparison with random walk. The work first looks at the behavior of first PCA components in random walk and compares it with the OU process. Then it looks at the DNNs and finds out that in the early phases of the training the behavior of weights are ballistic and later on they resemble more of an OU like process. The DNN part of the study is quite empirical but derivations are provided for the random walk.  Recently, there have been several works on understanding the landscape of loss functions and dynamics on it. The flatness of the minima may imply diffusive behavior at the bottom of the landscape which is consistent with the findings in this paper as well (generalization connection: https://arxiv.org/abs/1609.04836, local geometry at the bottom (for instance discussion in 298-304 fits into this framework): https://arxiv.org/abs/1803.06969, connecting solutions at the bottom: https://arxiv.org/abs/1806.06977, diffusive (on non-glassy loss surface) dynamics at the bottom: https://arxiv.org/abs/1803.06969, PCA on NNs: https://arxiv.org/abs/1602.07320, etc...). It's pity that the current work doesn't discuss the current literature, but well, ML research these days can lack many things including scrutiny in experiments and I rather see a methodologically sound work than the one with missing core discussion.   Speaking of the discussion, the paragraph between lines 22-30 has the correct conclusion, but the two papers cited there actually give different take-home messages. Dauphin et al is worried about the saddle points problem, and Choromanska et al is worried that the landscape will be mapped to a Hamiltonian of a spin glass (which is not true). The empirical results on finding the same loss values no matter what is interesting, but similar experiments can be found in several other works that do not necessarily claim that the landscape is glassy.   Regarding the claims of the referred papers in lines 48-51, all three papers need to use a version of the functional central limit theorem which is very hard to justify in the case of DNN where the number of parameters is huge (for example, for reasons mentioned in lines 206-209!). This debate on whether SGD is Langevin type dynamics is a major debate and it is quite unclear that this is the case. It's another drawback of the paper that it just cites a few works for which that support their vague idea without any discussion in depth. Overall, I think the introduction and discussion part of the paper is quite poor and it can be misleading without precisely stating such pitfalls.   The point of the above comments is not to include a massive discussion on every area, but to reduce the certainty in the language by picking a few papers that fit the general discussion. The paper has only 12 citations and 5 of them are quite non-essential as I have discussed above. At this stage of ML research, this is acceptable only if the work shows what they find. It is totally OK to show the results of their findings with minimal discussion, and the work will have its own value as a solid empirical work which is quite interesting and will have an impact in the community. But it is not OK to attempt to have a discussion with the missing sides of the story.    A further note on experiments: It is appreciated that the experiments are carried on in large-scale models with real data. However, it would be great to see it in a small fully connected model, as well. Say on MNIST and 2 hidden layer fully connected model, how wide should the network be in order to see this behavior consistently? If it is claimed to work on ResNets, there must be a regime for which it works on FCs. Another advantage of such a model is that one won't need to pick a layer in the middle and can do PCA on the full weight vector.   Also, for the ResNet, why only the sixth convolutional layer? Do you see this behaviour consistently in early layers and final layers as well? If not, why do you think that this is the case?   Overall, I think the direction of the paper is great but it would have been much richer with further discussions, it would've been less misleading with fewer discussions. It would've been more complete with more experiments even if, say, the claim doesn't work for the last layers, for example. It is a good paper, but I think there is room for crucial improvement in the discussion section (either by expansion or removal), and further room for intermediate complexity experiments. (And of course, it would be great to have further information on how exactly the experiments are conducted for reproducibility...)


Breaking the Activation Function Bottleneck through Adaptive Parameterization
Standard neural network architectures are non-linear only by virtue of a simple element-wise activation function, making them both brittle and excessively large. In this paper, we consider methods for making the feed-forward layer more flexible while preserving its basic structure. We develop simple drop-in replacements that learn to adapt their parameterization conditional on the input, thereby increasing statistical efficiency significantly. We present an adaptive LSTM that advances the state of the art for the Penn Treebank and WikiText-2 word-modeling tasks while using fewer parameters and converging in less than half the number of iterations.

The work proposes a form of adaptive parameterization for the activation function in neural nets. Several instantiations for feed-forward layers and a delicate adaptive design for LSTM are presented. Empirically, the adaptive version with few parameters outperforms the static version for MNIST classification and word-level language modeling.   While the idea of using adaptive parameterization is not new, the work introduces the concept of activation function bottleneck, which is not rigorously defined. If I understand correctly, the bottleneck conceptually refers to the fact that since the non-linearity is a static function of the pre-activation, in order to well exploit the "non-linear scaling" effect of the non-linearity, the weight matrix has to additionally encode some input specific information. As a result, the capacity of the weight matrix is not used efficiently. To break the bottleneck, one can make the weight matrix input specific (HyperNetworks) or the entire layer input specific in a more complicated way.   This work proposes a particular input-dependent parameterization of a layer by (1) removing the nonlinearity and (2) interleaving adaptive diagonal matrices produced by hyper-networks and static weight matrices. As the parameterization has good empirical performance, there are a few issues: - The proof of Property 1 is not satisfying, as it essentially assumes pi(x) can approximate G x / W.  - Further, while the original nonlinearity in a layer is removed, there are nonlinearities in the HyperNetworks. Hence, I tend to feel that the proposed adaptive layer is more like a more complicated sub-network with "multiplicative interactions" between various components, where some of these components are affine mappings of the input and others are non-linear mappings of the input. Thus, I'm not quite convinced by the perspective that the proposed layer structure is just an adaptive parameterization of the activation function.  In addition, for the Penn Treebank experiment, the performance improvement of the aLSTM with 17M parameters is relatively small compared to the improvement of the aLSTM with comparable parameter size in the wikitext-2 experiment. So, it's better to run an aLSTM with comparable parameter size to AWD-LSTM for Penn Treebank, which is more informative.  In summary, this work hypothesizes a "capacity bottlenck" due to the static behavior of the activation function without a formal definition. Notably, an effective adaptive parameterization of LSTM is proposed. However, no detailed analysis is given to justify the effectiveness of the proposed method, leaving the source of the effectiveness not clear.


Post: Device Placement with Cross-Entropy Minimization and Proximal Policy Optimization
Training deep neural networks requires an exorbitant amount of computation resources, including a heterogeneous mix of GPU and CPU devices. It is critical to place operations in a neural network on these devices in an optimal way, so that the training process can complete within the shortest amount of time. The state-of-the-art uses reinforcement learning to learn placement skills by repeatedly performing Monte-Carlo experiments. However, due to its equal treatment of placement samples, we argue that there remains ample room for significant improvements. In this paper, we propose a new joint learning algorithm, called Post, that integrates cross-entropy minimization and proximal policy optimization to achieve theoretically guaranteed optimal efficiency. In order to incorporate the cross-entropy method as a sampling technique, we propose to represent placements using discrete probability distributions, which allows us to estimate an optimal probability mass by maximal likelihood estimation, a powerful tool with the best possible efficiency. We have implemented Post in the Google Cloud platform, and our extensive experiments with several popular neural network training benchmarks have demonstrated clear evidence of superior performance: with the same amount of learning time, it leads to placements that have training times up to 63.7% shorter over the state-of-the-art.

This is a great work as it tackles an important problem: graph partitioning in heterogeneous/multi-device settings. There is an increasing number of problems that could benefit from resource allocation optimization techniques such as the one described in this work.  ML and specifically RL techniques have been recently developed to solve the problem of  device placement. This work addresses one of the main deficiencies of the prior work by making more sample efficient (as demonstrated by empirical results).   The novelty is in the way the placement parameters are trained: As oppose to directly train a placement policy for best runtime,  a softmax is used to model the distribution of op placements on devices (for each device among the pool of available devices.) At each iteration, the current placement distribution is refined by the conditional distribution representing the top 10% of the best placements. A KL-divergence loss is applied to minimize the distance between the old and new distribution. PPO is also used for further data sample reuse efficiency. The strengths: Great results: The results outperform existing approach based on Policy gradient by upto 58%.  The overhead of placement is low given that only 2400 samples were needed to achieve the results. This makes this work pottentially deployable.  Control experiments show improvement as a result of both KL minimization and PPO optimization. The results also show a clear win over prior policy gradient based work.  The paper is clear and easy to follow. The code is provided and looks very modular and easy to use.   Questions/suggestions:   What if the sampled placements from the conditional distribution at time t+1 produces placements that are all worse than the placements produces at iteration t? Has it ever happened in your experiments?  It would be good to show results on the amount of saving the optimized placements bring to the end-to-end training of the target models.  Th proposed model does not seem to generalize across graphs as it uses a one hot representation for nodes. Have the authors considered ways to incorporate graph embeddings into the model?  The reductions shows in Table 1 should compare Post with best baseline results, not with policy gradient.


Stochastic Nested Variance Reduction for Nonconvex Optimization
We study finite-sum nonconvex optimization problems, where the objective function is an average of n nonconvex functions. We propose a new stochastic gradient descent algorithm based on nested variance reduction. Compared with conventional stochastic variance reduced gradient (SVRG) algorithm that uses two reference points to construct a semi-stochastic gradient with diminishing variance in each iteration, our algorithm uses K + 1 nested reference points to build a semi-stochastic gradient to further reduce its variance in each iteration. For smooth nonconvex functions, the proposed algorithm converges to an ✏-approximate first-order stationary point (i.e., krF (x)k 2  ✏) within e O(n^✏ 2 + ✏ 3^n1/2 ✏ 2 ) 1 number of stochastic gradient evaluations. This improves the best known gradient complexity of SVRG O(n + n 2/3 ✏ 2 ) and that of SCSG O(n^✏ 2 + ✏ 10/3^n2/3 ✏ 2 ). For gradient dominated functions, our algorithm also achieves better gradient complexity than the state-of-the-art algorithms. Thorough experimental results on different nonconvex optimization problems back up our theory.

The paper proposes a stochastic nested variance reduced gradient descent method for non-convex finite-sum optimization. It has been studied that variance reduction in stochastic gradient evaluations improves the complexity of stochastic gradient evaluations. A popular method is stochastic variance reduced gradient (SVRG), which uses a single reference point to evaluate the gradient. Inspired by this, authors introduce variance reduction using multiple reference points with nested scheme. More precisely, each reference point updates in every T steps and the proposed algorithm uses K points and hence one-epoch iterates T^K loops. Authors demonstrate that gradient complexity of their algorithm is better than the original SVRG and stochastically controlled stochastic gradient (SCSG) under mild conditions. They also provide an algorithm and analysis for gradient dominated case, where the global optimal can be found. Finally, their algorithms show better results for practical nonconvex problems.  Although the idea of this paper seems to be natural, its result is very important in the sense of good theoretical analysis and experimental justifications. This paper is well written and easy to read. Hence, I believe that this paper should be accepted in NIPS.  One confusion is that it would be better if authors provide running time plot in experiments. Some readers may not satisfy with epochs-error plots. It makes this work to be more strengthen.   ***** I have read the author feedback. I appreciate authors for providing the details about my questions. *****


A Unified Feature Disentangler for Multi-Domain Image Translation and Manipulation
We present a novel and unified deep learning framework which is capable of learning domain-invariant representation from data across multiple domains. Realized by adversarial training with additional ability to exploit domain-specific information, the proposed network is able to perform continuous cross-domain image translation and manipulation, and produces desirable output images accordingly. In addition, the resulting feature representation exhibits superior performance of unsupervised domain adaptation, which also verifies the effectiveness of the proposed model in learning disentangled features for describing cross-domain data.

Paper summary: The authors propose a VAE-GAN architecture for learning multi-domain image representations where the domain information is disentangled from the domain invariant information, while simultaneously generating images of all considered domains. A domain classifier D_v pushes the VAE encoder E to learn a domain-invariant embedding z by adversarial training. This embedding is then concatenated with a one-hot encoding v of the domain before being fed to the VAE decoder G, which is then able to reconstruct domain-appropriate images. The VAE is further paired with an adversarial real / fake classifier, which improves image sharpness, and a domain classifier, which promotes mutual information between the generated images and the domain encoding v. Experiments are performed on human face generation and semi-supervised digits classification.  Comments: The proposed approach is pretty straightforward when compared to recent works on image generation / feature disentanglement (which is good), and the "unified" nature of the proposed architecture must be appreciated. The paper is well written and easy to follow, although some technical details are not sufficiently clear from the text (point #2 below). Furthermore, in my opinion the authors tend to overstate their contributions w.r.t existing methods, in particular the one in [17] (point #1 below). Finally, the experimental evaluation on image generation is a bit lacking (point #3 below). My current vote is for rejection, but I'm willing to increase my score given satisfying answers to the following three main points:  1) In my opinion the novelty of the proposed method w.r.t. the one in [17] is quite limited. In fact, the proposed network can be directly derived from the E-CDRD on of [17] by simply merging the source and target branches and adding a domain adversarial classifier to the hidden representation. Can you please elaborate on this? Are there any other differences (theoretical or practical / architectural) that I'm missing? 2) I think that the overall clarity of the paper would benefit from moving part of the information reported in the supplementary material to the main paper. In particular, I find it a bit odd that nowhere in the paper is ever mentioned that v is a one-hot encoding of the domain labels, or that no information is ever given about the architectures of the various modules composing the network. 3) All the results shown on image generation are of a qualitative kind, and there's no comparison with existing methods. While, unfortunately, this is becoming alarmingly common in this line of research, I still think that authors should strive to evaluate as objectively as possible their proposed methods. In fact, it's not inconceivable that the generated images shown in the paper could be carefully selected outliers. Performing this kind of objective evaluation is a non-trivial tasks, but some de-facto standards currently exist, including Mechanical Turk-based perceptual evaluation (like in [3]), Inception score (like in REF1 below), or GAN convergence metrics. In the absence of any of these I would argue that, from the purely scientific point of view, the results in Section 4.2 are effectively meaningless.  REF1: Tim  Salimans,  Ian  Goodfellow,  Wojciech  Zaremba,  Vicki  Cheung,  Alec  Radford,  and  Xi  Chen. Improved techniques for training gans. In Neural Information Processing Systems (NIPS), 2016  --- Additional comments after author response phase --- The authors gave satisfactory answers to my three main points of criticism, although I still think that the novelty of the proposed method w.r.t the state of the art is a bit limited and should not be emphasized too much in the text. Overall, I think the paper can be accepted.


A Theory-Based Evaluation of Nearest Neighbor Models Put Into Practice
In the k-nearest neighborhood model (k-NN), we are given a set of points P , and we shall answer queries q by returning the k nearest neighbors of q in P according to some metric. This concept is crucial in many areas of data analysis and data processing, e.g., computer vision, document retrieval and machine learning. Many k-NN algorithms have been published and implemented, but often the relation between parameters and accuracy of the computed k-NN is not explicit. We study property testing of k-NN graphs in theory and evaluate it empirically: given a point set P ⊂ R δ and a directed graph G = (P, E), is G a k-NN graph, i.e., every point p ∈ P has outgoing edges to its k nearest neighbors, or is it -far from being a k-NN graph? Here, -far means that one has to change more than an -fraction of the edges in order to make G a k-NN graph. We develop a randomized algorithm with one-sided error that decides this question, i.e., a property tester for the k-NN property, with complexity O( √ nk 2 / 2 ) measured in terms of the number of vertices and edges it inspects, and we prove a lower bound of Ω( n/ k). We evaluate our tester empirically on the k-NN models computed by various algorithms and show that it can be used to detect k-NN models with bad accuracy in significantly less time than the building time of the k-NN model.

SUMMARY: The paper studies the problem of testing whether a graph is epsilon-far from a kNN graph, where epsilon-far means that at least epsilon-fraction of the edges need to be changed in order to make the graph a kNN graph.  The paper presents an algorithm with an upper bound of O(\sqrt{n}*k^2/\epsilon^2) number of edge/vertex queries and a lower bound of \Omega(\sqrt{n}).   COMMENTS:  1. Ln 175. I guess "\omega" should be "p"  2. The result of Proposition 12 is interesting which bounds the number of points that can be in the kNN set of a particular point. The bound is k times the known bound for 1-NN. I wonder if this could be tightened somehow. Also, are there lower bounds for the 1-NN result?  3. Is there a reason why the \epsilon is hidden from the lower bound?  4. It looks like these result don't explicitly show the dependence on the dimension of the space. I guess similar comments about the lower bound: could we get some lower bounds that depend on the dimension of the space? Currently the lower bound is in 1-dimension. I guess we will have some factor that depends exponentially in the dimension of the space.  5. In practice, some mistakes are more costly than others. For example, deleting the edge to the k-th nearest neighbor and adding in an edge to the (k+1)-th nearest neighbor is not as bad as deleting the edge to the k-th nearest neighbor and adding an edge to the farthest point. Can the analysis be extended to take this into account and e.g. have some weighted cost?   OVERALL: Testing whether a graph is approximately a kNN graph is an interesting problem with implications to approximate nearest neighbor (ANN) procedures. The paper provides an algorithm which tests whether such an ANN index actually approximates the exact kNN method which has reasonable computational cost which could have applications in cheaply testing whether an ANN index is acceptable for usage. The algorithm and theoretical results presented are to my knowledge novel. These results come with some experimental results on a number of datasets which suggest that this may have relevence in practice.


The Convergence of Sparsified Gradient Methods
Stochastic Gradient Descent (SGD) has become the standard tool for distributed training of massive machine learning models, in particular deep neural networks. Several families of communication-reduction methods, such as quantization, largebatch methods, and gradient sparsification, have been proposed to reduce the overheads of distribution. To date, gradient sparsification methods-where each node sorts gradients by magnitude, and only communicates a subset of the components, accumulating the rest locally-are known to yield some of the largest practical gains. Such methods can reduce the amount of communication per step by up to three orders of magnitude, while preserving model accuracy. Yet, this family of methods currently has no theoretical justification. This is the question we address in this paper. We prove that, under analytic assumptions, sparsifying gradients by magnitude with local error correction provides convergence guarantees, for both convex and non-convex smooth objectives, for data-parallel SGD. The main insight is that sparsification methods implicitly maintain bounds on the maximum impact of stale updates, thanks to selection by magnitude. Our analysis also reveals that these methods do require analytical conditions to converge well, justifying and complementing existing heuristics.

The paper proposes a theoretical study of a heuristic method for distributed SGD, using the sparsification gradients techniques.  The technique is not new, but this theoretical study is, in particular authors claim that their analysis does not require a bound on the maximum delay, which departs from already known analysis for asynchronous variants of SGD. I am however not confortable with the main assumption of the paper (Assumption 1), which is not discussed enough in my opinion, or numerically checked in experiments. It is I believe a strong assumption, that provides an ad-hoc control of the TopK operator. A comparison with the bounded delay assumption for asynchronous SGD is not fair, since this assumption is usually satisfied / numerically checked in experiments, while it is hardly the case of Assumption 1 for the TopK SGD. I think that the numerical experiments are weak : the comparison with asynchronous SGD is on a convex problem only (while the paper markets the use of TopK SGD for deep nets) and only illustrate marginal improvements. As a conclusion, I find the paper interesting, and won't oppose its acceptance, but I think that it is only marginally above the acceptance threshold for the NIPS venue.


Statistical mechanics of low-rank tensor decomposition
Often, large, high dimensional datasets collected across multiple modalities can be organized as a higher order tensor. Low-rank tensor decomposition then arises as a powerful and widely used tool to discover simple low dimensional structures underlying such data. However, we currently lack a theoretical understanding of the algorithmic behavior of low-rank tensor decompositions. We derive Bayesian approximate message passing (AMP) algorithms for recovering arbitrarily shaped low-rank tensors buried within noise, and we employ dynamic mean field theory to precisely characterize their performance. Our theory reveals the existence of phase transitions between easy, hard and impossible inference regimes, and displays an excellent match with simulations. Moreover it reveals several qualitative surprises compared to the behavior of symmetric, cubic tensor decomposition. Finally, we compare our AMP algorithm to the most commonly used algorithm, alternating least squares (ALS), and demonstrate that AMP significantly outperforms ALS in the presence of noise.
AMP algorithms have a long history dating back to early work on the statistical physics of perceptron learning [7,8] (see [9] for a review). The term AMP was coined by Donoho, Maleki and Montanari in their work on compressed sensing [10] (see also [11,12,13,14,15,16] for replica approaches to compressed sensing and high dimensional regression). AMP approximates belief propagation in graphical models and a rigorous analysis of AMP was carried out in [17]. For a rank-one matrix estimation problem, AMP was first introduced and analyzed in [18]. This framework has been extended in a beautiful body of work by Krzakla and Zdeborova and collaborators to various lowrank matrix factorization problems in [19,20,21,22]). Recently low-rank tensor decomposition through AMP was studied in [21], but their analysis was limited to symmetric tensors which are then necessarily cubic in shape. In [23], a similar approach was used to extend the analysis of order-2 tensors (matrices) to order-3 tensors, which can potentially be further extended to higher orders.
However, tensors that occur naturally in the wild are almost never cubic in shape, nor are they symmetric. The reason is that the p different modes of an order p tensor correspond to measurements across very different modalities, resulting in very different numbers of dimensions across modes, yielding highly irregularly shaped, non-cubic tensors with no symmetry properties. For example in EEG studies 3 different tensor modes could correspond to time, spatial scale, and electrodes [24]. In fMRI studies the modes could span channels, time, and patients [25]. In neurophysiological measurements they could span neurons, time, and conditions [26] or neurons, time, and trials [3]. In studies of visual cortex, modes could span neurons, time and stimuli [27].
Thus, given that tensors in the wild are almost never cubic, nor symmetric, to bridge the gap between theory and experiment, we go beyond prior work to derive and analyze Bayes optimal AMP algorithms for arbitrarily shaped high order and low rank tensor decomposition with different priors for different tensor modes, reflecting their different measurement types. We find that the low-rank decomposition problem admits two phase transitions separating three qualitatively different inference regimes: (1) the easy regime at low noise where AMP works, (2) the hard regime at intermediate noise where AMP fails but the ground truth tensor is still possible to recover, if not in a computationally tractable manner, and (3) the impossible regime at high noise where it is believed no algorithm can recover the ground-truth low rank tensor.
From a theoretical perspective, our analysis reveals several surprises relative to the analysis of symmetric cubic tensors in [21]. First, for symmetric tensors, it was shown that the easy inference regime cannot exist, unless the prior over the low rank factor has non-zero mean. In contrast, for non-symmetric tensors, one tensor mode can have zero mean without destroying the existence of the easy regime, as long as the other modes have non-zero mean. Furthermore, we find that in the space of all possible tensor shapes, the hard regime has the largest width along the noise axis when the shape is cubic, thereby indicating that tensor shape can have a strong effect on inference performance, and that cubic tensors have highly non-generic properties in the space of all possible tensor shapes.
Before continuing, we note some connections to the statistical mechanics literature. Indeed, AMP is closely equivalent to the TAP equations and the cavity method [28,29] in glassy spin systems. Furthermore, the posterior distribution of noisy tensor factorization is equivalent to p-spin magnetic systems [30], as we show below in section 2.2. For Bayes-optimal inference, the phase space of the problem is reduced to the Nishimori line [31]. This ensures that the system does not exhibit replicasymmetry breaking. Working in the Bayes-optimal setting thus significantly simplifies the statistical analysis of the model. Furthermore, it allows theoretical insights into the inference phase-transitions, as we shall see below. In practice, for many applications the prior or underlying rank of the tensors are not known a-priori. The algorithms we present here can also be applied in a non Bayes-optimal setting, where the parametric from of the prior can not be determined. In that case, the theoretical asymptotics we describe here may not hold. However, approximate Bayesian-optimal settings can be recovered through parameter learning using expectation-maximization algorithms [32]. We discuss these consequences in section 4. Importantly,the connection to the statistical physics of magnetic systems allows the adaptation of many tools and intuitions developed extensively in the past few decades, see e.g. [33]. We discuss more connections to statistical mechanics as we proceed below.

Summary: Aim of the paper is to provide a better theoretical understanding of the approximate message passing (AMP) algorithm for low-rank tensor decomposition. For this method, the author show that a mean field approximation may be used to reveal the existence of different regimes (phases) where the inference is easy, hard or impossible. They also extend the AMP algorithm of previous works to the case of non-symmetric tensors.  Strengths: Low-rank tensor decomposition is a widespread and hard problem. Defining regimes where inference is possible/impossible may help obtain one a better high-level interpretation of the task. The analysis proposed by the authors is theoretically sound and connects the tensor factorization task with nice concepts from the physics literature. Such connections have been shown to be useful and inspiring in many other cases in computer science. Finally, the proposed algorithm considerably extends the applicability of the AMP approach.  Weaknesses: It looks complicated to assess the practical impact of the paper. On the one hand, the thermodynamic limit and the Gaussianity assumption may be hard to check in practice and it is not straightforward to extrapolate what happens in the finite dimensional case. The idea of identifying the problem's phase transitions is conceptually clear but it is not explicitly specified in the paper how this can help the practitioner. The paper only compares the AMP approach to alternate least squares without mention, for example, positive results obtained in the spectral method literature. Finally, it is not easy to understand if the obtained results only regard the AMP method or generalize to any inference method.     Questions: - Is the analysis restricted to the AMP inference? In other words, could a tensor that is hard to infer via AMP approach be easily identifiable by other methods (or the other way round)?   - Are the easy-hard-impossible phases be related with conditions on the rank of the tensor?  - In the introduction the authors mention the fact that tensor decomposition is in general harder in the symmetric than in the non-symmetric case. How is this connected with recent findings about the `nice' landscape of the objective function associated with the decomposition of symmetric (orthogonal) order-4 tensors [1]?  - The Gaussian assumption looks crucial for the analysis and seems to be guaranteed in the limit r << N. Is this a typical situation in practice? Is always possible to compute the `effective' variance for non-gaussian outputs? Is there a finite-N expansion that characterize the departure from Gaussianity in the non-ideal case? - For the themodynamic limit to hold, should one require N_alpha / N  =  O(1) for all alpha? - Given an observed tensor, is it possible to determine the particular phase it belongs to?  [1] Rong Ge and Tengyu Ma, 2017, On the Optimization Landscape of Tensor Decompositions