Natasha 2: Faster Non-Convex Optimization Than SGD
We design a stochastic algorithm to find ε-approximate local minima of any smooth nonconvex function in rate O(ε −3.25 ), with only oracle access to stochastic gradients. The best result before this work was O(ε −4 ) by stochastic gradient descent (SGD).

This submission considers an online optimization problem, where the underlying objective function is either an expectation or a finite sum of functions. It is assumed that the objective is nonconvex, and the goal of the submission is to develop an efficient online algorithm (that is, a method with a complexity that is independent on the number of components defining the objective) that finds an approximate local minimum, i. e. a point at which the gradient norm is less than $\epsilon$ and the minimum Hessian eigenvalue is at least $-\delta$, where $\epsilon$ and $\delta$ are two positive thresholds. The development of online algorithms is extremely relevant in the context of machine learning, in that it reduces the dependence of the method to the size of the available data. Considering nonconvex problems is also of critical importance, particularly given the outbreak of deep learning formulations.  The submission appears to be written in order to help the reader follow the reasoning that lead to the derivation of the new method and results. The introductory section conveys in my opinion the right ideas, for its most part. I am concerned by the comparison made on lines 71-76: it seems that the best guarantees obtained in references [35], [7,39] and this submission correspond to different choices for the second-order tolerance $\delta$, namely, $\delta=\epsilon^{1/2}$ for [35] (and [39] for a bound in $\epsilon^{-3.5}$), $\delta=\epsilon^{4/9}$ for [39] and a bound in $\epsilon^{-3.333}$, and $\delta=\epsilon^{1/4}$ for [7] and this submission, with bounds respectively in $\epsilon^{-3.333}$ and $\epsilon^{-3.25}$  Adding this precision would help in comparing those methods, especially given that some of the cited references study multiple algorithms (this is also valid for Table 1 in appendix, to a lesser extent).  The core of the submission begins in Section 3 with the description of the Natasha 1.5 method to find approximate stationary points, at which the gradient has small norm. This method is an extension of the original Natasha algorithm, adapted to the online setting by exploiting a so-called ``strong nonconvexity" property of the function. Certain aspects and remarks of this section can be difficult to understand without resorting to the appendix: in particular, the use of a proximal operator (and Definition 2.2) may be deferred to the appendix, and more simply mentioned, in order to focus on the exploitation of the strong nonconvexity property, and its main difference with the original Stochastically Controlled Stochastic Gradient (SCSG) paradigm.   The main algorithm, Natasha 2, is detailed in Section 4. The conditions $\delta \in (0, \epsilon^{1/4})$ in Theorem 2 and $\delta \ge \epsilon^{1/4}$ in Section 1 (or $\delta > \epsilon^{1/4}$ in Remark 4.4) seem mutually exclusive, and I believe the presentation would have been simplified by considering only the special case $\delta = \epsilon^{1/4}$. Referring to the appendix is also helpful here to understand what is behind these informal theorems, particular toward the end of the section as several technical difficulties are sketched.  Overall, even if the results and the analysis appear correct to me, it is my opinion that the presentation of the submission can prevent a reader from fully appreciating the quality of its results. More specifically, the novelty of the Natasha 2 algorithm, that allows for the derivation of a bound in $\epsilon^{-3.25}$,  appears hidden among the various considerations mentioned in the submission, including the fact that this bound is not necessarily better than those in $\epsilon^{-3.5}$. Moreover, the submission relies heavily on its supplementary material, which tends to indicate that the main body is not sufficient to get a good idea of this work. I strongly believe that the results of the submission are sufficiently original and significant: it is clear that they have inspired follow-up works, and in that sense they fully deserve publication. However, it does not seem that the NIPS format is the most appropriate for exposing the method, and in my opinion, it would not do justice to the efforts made by the authors to position their work within the existing literature.  For these reasons, I regretfully give this submission an overall score of 4 for my initial review.  Additional comments:  a) Some sentences are oddly formulated, for instance:   - ``Being easy to implement, however, we raise two main efficiency issues" line 33; - ``any point close to a saddle point must have a negative curvature"  lines 53-54.  b) The derivation of the series of equations lines 674-675 could be more detailed. There is also an extra $p$ on the last three equations of line 674, as the term is in the sum.  Edit of the review following author feedback:  I am pleased that the author took my review into consideration, and I concur with his responses. In my opinion, addressing these concerns was necessary to properly position this work into the literature (and acknowledge its impact on later works). I trust the author to make the necessary changes to his submission so as to address these concerns, and correct the typos that I pointed out.   One exception is my comment about simplifying the presentation by choosing $\delta=\epsilon^{1/4}$ throughout the submission. I understand the author's point in the feedback, and although I do believe it would make a camera-ready version easier to understand, I will not insist on it. The author should feel free to maintain the use of $\delta$.  Even though it seems to me that this submission has grown out beyond a NIPS paper ever since its first appearance online, I am willing to change my assessment and recommend acceptance of the submission, following the other reviewers' assessments.


Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
We marry two powerful ideas: deep representation learning for visual recognition and language understanding, and symbolic program execution for reasoning. Our neural-symbolic visual question answering (NS-VQA) system first recovers a structural scene representation from the image and a program trace from the question. It then executes the program on the scene representation to obtain an answer. Incorporating symbolic structure as prior knowledge offers three unique advantages. First, executing programs on a symbolic space is more robust to long program traces; our model can solve complex reasoning tasks better, achieving an accuracy of 99.8% on the CLEVR dataset. Second, the model is more data-and memory-efficient: it performs well after learning on a small number of training data; it can also encode an image into a compact representation, requiring less storage than existing methods for offline question answering. Third, symbolic program execution offers full transparency to the reasoning process; we are thus able to interpret and diagnose each execution step.

The paper studies the combination of deep representation learning for visual recognition and language understanding, and symbolic program execution for reasoning.  The focus is on the development of a Visual Question Answering System, where question related to a scene are posted to the system. The idea is quite simple to understand: the system first recovers information related to the structure of the scene and represents it symbolically, together with the symbolic representation (in the form of a program) of the question. Then, the program is executed over the representation of the scene and the answer is computed.  The first step is done by means of machine learning techniques for visual recognition and NLP techniques (when the question is posed in NL) to obtain the program associated to the question. The learned knowledge is then represented symbolically which provides the ability of a higher level of reasoning about the scene. The authors then show experimentally that the proposed system has a high percentage of accuracy on a benchmark dataset. They also show that it is more efficient in data and memory usage (the symbolic abstraction helps in reasoning and learning with fewer examples, as humans do). The authors state that to the best of their knowledge this is the first work that makes this combination allowing also for transparency and explanations in the reasoning process. To the best of my knowledge this true, but as this is not my area of expertise cannot be completely sure.  The idea is novel, is timely, and sound to the extent I checked. The authors hypothesis is that machine learning techniques is not enough to perform high level reasoning tasks (as question answering) over visual scenes. A way to incorporate and structure context knowledge is necessary. This proposal achieves exactly that.  My only concern is that the representation language used is not clearly defined. There is no formal definition or specification of what is a symbolic program. What language is used? What is a symbolic module? I this sense I find the work still behind of what symbolic reasoning is supposed to involve.  Author's Feedback: the authors have answered satisfactorily my concerns.


(Probably) Concave Graph Matching
In this paper, we address the graph matching problem. Following the recent works of Zaslavskiy et al. (2009);Vestner et al. (2017) we analyze and generalize the idea of concave relaxations. We introduce the concepts of conditionally concave and probably conditionally concave energies on polytopes and show that they encapsulate many instances of the graph matching problem, including matching Euclidean graphs and graphs on surfaces. We further prove that local minima of probably conditionally concave energies on general matching polytopes (e.g., doubly stochastic) are with high probability extreme points of the matching polytope (e.g., permutations).

Update: I have considered the author response. Some of my previous questions have been answered. I have updated my review and "overall score".  ==========================================================  This paper focuses on graph matching problems with quadratic objective functions. These objectives are usually equivalent to a quadratic assignment problem (QAP) [1] on the permutation matrix space. The authors focused on the Frank-Wolfe algorithm (Algorithm 1) on the objectives with the relaxation from the permutation matrix space to the doubly stochastic matrix space DS. To analyze the performances of this algorithm, the authors defined conditionally concave (or negative definite) and used these definitions to do some theoretical analysis on the local optima convergence of this algorithm. Then the authors extended this definition to a probabilistic setting and did some analysis for a more general form of graph matching problems. At the end, the authors provided another algorithm (Algorithm 2) for the one sided permutation problems where the probably conditionally concavity does not hold.  Graph matching problems are very popular now in the machine learning community. In my opinion, the authors proposed some very interesting and novel definitions, and these definitions may be very useful for doing relaxations for graph matching problems. However, it will be better if more analysis on the optimality of the convergence local optima can be shown instead of just showing some sufficient conditions for the algorithm to guarantee (or with high probability) to converge to extreme point local optima.   Strengths of this paper:  1. Proposed the definition for (probably) conditionally concave and (probably) conditionally negative definite. These definitions can clearly capture the possibility for the Frank-Wolfe algorithm to locally optimize the objective function.  2. The theoretical analysis on these new concavity definitions provide us important insight on the convergence performances of Algorithm 1, under certain circumstances.  Weakness:  1. As I just mentioned, the paper only analyzed, under which cases will the Algorithm 1 converges to permutations as local minima. However, it will be better if the quality of this kind of local minima could be analyzed (e.g. the approximation ratio of these local minima, under certain assumptions).  2. This paper is not very easy to follow. First, many definitions are used before they are defined. For example, on line 59 in Theorem 1, the authors used the definition of  "conditionally positive definite function of order 1", which is defined on line 126 in Definition 2. Also, the author used the definition "\epsilon-negative definite" on line 162, which is defined on line 177 in Definition 3. It will be better if the authors could define those important concepts before using them. Second, the introduction is a little bit too long (more than 2.5 pages) and many parts of that are repeated in Section 2 and 3. It might be better to restructure the first 3 sections for a little bit. Third, it will be good if more captions could be added to the figures in the experiment section so that the readers could understand the results more easily.  3. For the description of Theorem 3, from the proof it seems that we need to use Equation 12 as a condition. It will be better if this information is included in the theorem description to avoid confusions (though I know this is mentioned right before the theorem, it is still better to have it in the theorem description). Also, for the constant c_1, it is better to give it an explicit formula in the theorem.   Reference:  [1] Burkard, R. E., Cela, E., Pardalos, P. M., and Pitsoulis, L. S. (1998). The quadratic assignment problem. In Handbook of combinatorial optimization, pages 1713–1809. Springer.


Adaptive Negative Curvature Descent with Applications in Non-convex Optimization
Negative curvature descent (NCD) method has been utilized to design deterministic or stochastic algorithms for non-convex optimization aiming at finding second-order stationary points or local minima. In existing studies, NCD needs to approximate the smallest eigen-value of the Hessian matrix with a sufficient precision (e.g., 1) in order to achieve a sufficiently accurate second-order stationary solution (i.e., λ min (∇ 2 f (x)) ≥ − 2 ). One issue with this approach is that the target precision 2 is usually set to be very small in order to find a high quality solution, which increases the complexity for computing a negative curvature. To address this issue, we propose an adaptive NCD to allow an adaptive error dependent on the current gradient's magnitude in approximating the smallest eigen-value of the Hessian, and to encourage competition between a noisy NCD step and gradient descent step. We consider the applications of the proposed adaptive NCD for both deterministic and stochastic non-convex optimization, and demonstrate that it can help reduce the the overall complexity in computing the negative curvatures during the course of optimization without sacrificing the iteration complexity.

This paper utilizes the Negative curvature descent (NCD) method to design adaptive algorithms for non-convex optimization aiming at finding second-order stationary points or local minima. The theoretical convergence rate of the adaptive NCD matches the state-of-the-art NCD-AG. Supporting experiment results using cubic regularization, regularized non-linear least-square, and one hidden-layer NN implies a significant supercedes the NCD-AG.  In some sense, this work is a continuation of the work by Carmon et al (2016) in the deterministic setting. This work made a key contribution that at each update of AdaNCD\textsuperscript{det}, it chooses one of the GD (gradient-descent) step and NCD (Negative curvature descent / Lanczos) step, whichever decreases the function value more. The benefit of this method is that one only need to solve NCD with an accuracy of $O(\sqrt{\epsilon} \lor \|\nabla f(x)\|^{1/2})$ instead of $O(\sqrt{\epsilon})$. It is unsurprising to me (and also argued in the paper) that the convergence rate matches the state-of-the-art $O(\epsilon^{-1.75})$ gradient complexity when finding an $(\epsilon,\sqrt{\epsilon})$-approximate SOSP, which is the state-of-the-art local-minimizer finding algorithm [Carmon et al 2016, Agarwal et al 2017].  I tend to vote yes for this paper, not because of its technical proof method which has largely appeared in earlier literatures, but because the simple adaptive twist that makes it numerically work in practice. Most non-convex optimization theory works notoriously lack of such numerical support, however, making them numerically unattended.  The work also dealt with the case when $\epsilon_2 = \epsilon^\alpha$ with $\alpha\in (0,1)$, and stochastic setting. Perhaps, in future work, the author also want to include the analysis of the finite-sum setting.   Some minor comments:  In Algorithm 5, the authors might want to explain why one can solve AdaNCG at accuracy $\epsilon^{3/4}$ and then switch gear to the AGD iteration. Should the accuracy be $\epsilon$ so they match up?  The authors might want to carefully name the algorithms so as not to cause any confusion. For example, Algorithm 1, 3 and 5 have similar names, but the second (resp.~third) serves as a subroutine of the first (resp.~second).  For the stochastic setting, the AdaNCD+SCSG result seems very similar to the so-called Neon+SCSG method (Xu et al 2017; Allen Zhu and Li, 2017). Does the complexity in Theorem 4 matches up with Neon2 in Allen-Zhu and Li (2017)?


Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo
Deep Gaussian Processes (DGPs) are hierarchical generalizations of Gaussian Processes that combine well calibrated uncertainty estimates with the high flexibility of multilayer models. One of the biggest challenges with these models is that exact inference is intractable. The current state-of-the-art inference method, Variational Inference (VI), employs a Gaussian approximation to the posterior distribution. This can be a potentially poor unimodal approximation of the generally multimodal posterior. In this work, we provide evidence for the non-Gaussian nature of the posterior and we apply the Stochastic Gradient Hamiltonian Monte Carlo method to generate samples. To efficiently optimize the hyperparameters, we introduce the Moving Window MCEM algorithm. This results in significantly better predictions at a lower computational cost than its VI counterpart. Thus our method establishes a new state-of-the-art for inference in DGPs.

The paper addresses the problem of doing inference in deep Gaussian processes (DGPs). The authors propose to use Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) as a way to relax the assumption of Gaussianity for the inducing point posterior as assumed in the standard variational inference setting. To motivate this, the authors argue based on empirical evidence that the posterior distributions of interest are not always Gaussian nor unimodal.  In order optimize the hyperparameters during training, they propose a new variant of the Markov Chain Expectation Maximization (MCEM) algorithm called Moving Window MCEM, which is claimed to perform better results than the original MCEM algorithm.  Finally, the authors demonstrate the performance ofthe proposed method on a set of benchmark datasets (UCI datasets, MNIST classification, and Harvard Clean Energy). The experiments show that the proposed method outperforms the benchmark methods.   The technical details of the submission appear sound and correct. While none of the claims in the paper is supported by any theoretical analysis, all claims are to some degree supported by empirical evidence. However, the claim that Moving Window MCEM performs better than MCEM is only supported by a plot of the training log likelihood vs run time for a single data set. Similarly, the claim that SGHMC converges faster than the reference method (DSVI) is also only supported by a plot of the test log likelihood vs run time for a single data set. However, the experiments based on the UCI datasets do show that the proposed method achieved better predictive performance than the benchmark methods, especially for the larger datasets.   The paper is in general clear, well-organized, and well-written. However, the proposed method has many moving parts (SGHMC, the autotuning method for the SGHMC parameters and the Moving Window MCEM) and it’s not entirely clear how each of these is executed relative to each other. Therefore, it would be an improvement to include pseudo-code for the entire method in the supplementary material and not just for the Moving Window EM algorithm.  Furthermore, there are a few statements that could be made more precise, see comments below.   The combination of SGHMC and deep Gaussian processes is novel. Furthermore, the proposed variant of the MCEM is also original.   Constructing deep models that are capable of quantifying uncertainty in a reasonable way is an important topic. Therefore, it is most likely that other researchers will use the proposed ideas and improve them. Additionally, the proposed method achieves state of the art performance in terms of predictive accuracy for a range of benchmark datasets and therefore practitioners will most likely also be interested in the method.   Further comments: Line 61: “One might expect a sampling method to be more computationally intensive than an approximate method such as DSVI.” Sampling-based methods are also approximate  Line 62: “However, in DGPs, sampling from the posterior is inexpensive, since it does not require the recomputation of the inverse covariance matrix, which only depends on the Hyperparameters.” The authors should clarify if the is always true or if it’s only true for the “sampling phase”, where the hyperparameters are kept fixed  Line 82: “The probability of y...” This is not a meaningful quantity - the authors probably mean the conditional distribution of y   Line 88: “The GP distribution f given the inducing” typo  Line 126: “First, we illustrate with a toy problem that the posterior distribution in DGPs can be multimodal.” The authors state that there are two modes for the toy problem, but they should clarify of these modes are related to each other in any way. Furthermore, the plots in figure 1 only show f1 vs x and y vs f1, but it would be more interesting to see what y vs x looks like for the two modes.   Line 140: "We examine the posterior samples generated by SGHMC for each  inducing output" What measures have you taken to make sure that SGHMC have  actually converged to the target distributions? If you cannot guarantee that the samples from SGHMC are representative of the true target distributions, then these tests lose their value.  Line 194: Missing a log in the M-step?   Line 212: “We plotted the predictive log-likelihood on the training set against the runtime of the algorithm to demonstrate the superior  performance of Moving Window MCEM over MCEM (the test log-likelihood shows the same trend)” In that case, it would be more convincing to show the test log likelihood  After the rebuttal ------------------------------ I have read the rebuttal, but it does not change my opinion of the paper as the authors did properly address all of my concerns. For example, it would have been nice if the authors would have commented on the nature of the multimodality in terms of symmetries and what measures they have taken to guarantee that SGHMC produces samples from the true posterior distribution for their statistical significance tests.


Knowledge Distillation by On-the-Fly Native Ensemble
Knowledge distillation is effective to train the small and generalisable network models for meeting the low-memory and fast running requirements. Existing offline distillation methods rely on a strong pre-trained teacher, which enables favourable knowledge discovery and transfer but requires a complex two-phase training procedure. Online counterparts address this limitation at the price of lacking a high-capacity teacher. In this work, we present an On-the-fly Native Ensemble (ONE) learning strategy for one-stage online distillation. Specifically, ONE only trains a single multi-branch network while simultaneously establishing a strong teacher on-the-fly to enhance the learning of target network. Extensive evaluations show that ONE improves the generalisation performance of a variety of deep neural networks more significantly than alternative methods on four image classification dataset: CIFAR10, CIFAR100, SVHN, and ImageNet, whilst having the computational efficiency advantages.

Summary:  Authors propose a novel multi-branch network with a loss function that uses distillation from a combined branch to distill into individual branches. The technique is motivated by the idea that Teacher-Student knowledge distillation is a two-step process often requiring a large pre-trained teacher. Their method builds a teacher, out of weighted ensemble and uses that to train the network. They are able to show that the combined network (ONE-E) is far superior to standalone networks, and the individual branch (ONE) is also better than its counterpart (i.e if it were trained without any of the loss functions and the branches).  Pros: 1. Excellent write-up This is a very well written paper. Literature study is comprehensive and the whole paper is easy to follow even if it looks dense in contentwise.  2. Intuitive and excellent idea The idea presented is both intuitive and excellent. People have been using an ensemble of models for various tasks, and it is well known that ensemble will give better results as long as there is some variance in the model.  Another approach to improve the accuracy of the model is to train it with soft-logits of a bigger model. They were able to take advantage of both the techniques (ensemble and knowledge distillation) in a single setup.  3. Excellent ablation study They have presented their technique, as a network modification (adding branches, sharing layers) and change is an objective function (KL div loss + CE). They show the result of each of these techniques individually (Table 5) and show that the combination of all of these provides the best result. This leaves no doubt to the reader on the importance of each of the changes.  4. Superior results Their results on CIFAR/SVHN/Imagenet all look very good and promising. It looks like the technique improves the result for various types of model, thus it is agnostic to the structure (Table 1). It outperforms other ensemble techniques (Table 4) which are more computationally expensive than the presented ONE technique.  Cons:  1. Section 4.5 feels incomplete and an after-thought. I understand that space was a huge limitation, however, if possible, I would like to see a better analysis of why this works compared to other ensemble and Mixture of Expert techniques. Ensembles work when the models have a variance between them. There is nothing in the current setup to encourage that. Then why does it do better? In a 5 branch setup, how different are these branches? Do they make the same errors or different errors?  (Minor) 2. I would like to see more ImageNet results, especially with more efficient models like SENet.   3. How does ONE-E compare to Sparsely Gated MoE [1]?  4. It was not mentioned how many low-level layers where shared.   [1] https://arxiv.org/pdf/1701.06538.pdf


Parameters as interacting particles: long time convergence and asymptotic error scaling of neural networks
The performance of neural networks on high-dimensional data distributions suggests that it may be possible to parameterize a representation of a given highdimensional function with controllably small errors, potentially outperforming standard interpolation methods. We demonstrate, both theoretically and numerically, that this is indeed the case. We map the parameters of a neural network to a system of particles relaxing with an interaction potential determined by the loss function. We show that in the limit that the number of parameters n is large, the landscape of the mean-squared error becomes convex and the representation error in the function scales as O(n −1 ). In this limit, we prove a dynamical variant of the universal approximation theorem showing that the optimal representation can be attained by stochastic gradient descent, the algorithm ubiquitously used for parameter optimization in machine learning. In the asymptotic regime, we study the fluctuations around the optimal representation and show that they arise at a scale O(n −1 ). These fluctuations in the landscape identify the natural scale for the noise in stochastic gradient descent. Our results apply to both single and multi-layer neural networks, as well as standard kernel methods like radial basis functions.
proved for radial basis functions [16]. However, the proofs of the universal approximation theorems do not ensure that any particular optimization technique can locate the ideal set of parameters.
Parameters as particles-In order to study the properties of stochastic gradient descent for neural network optimization, we recast the standard training procedure in terms of a system of interacting particles [17]. In doing so, we give an exact rewriting of stochastic gradient descent as a stochastic differential equation with multiplicative noise, which has been studied previously [18,19]. We interpret the limiting behavior of the parameter optimization via a nonlinear Liouville equation for the time evolution of a parameter distribution [20]. This framework provides analytical tools to determine a Law of Large Numbers for the convergence of the optimization and to derive scaling results for the error term as time and the number of parameters grow large. A similar perspective has been adopted concurrently by Mei et al. [21], Chizat and Bach [22], and Sirignano and Spiliopoulos [23], which study the "mean field limit", similar to our Law of Large Numbers, but not asymptotic fluctuations or error scaling.
Convergence and asymptotic dynamics of stochastic gradient descent-We demonstrate that the optimization problem becomes convex in the limit n → ∞ and we show that both gradient descent and SGD convergence to the global minimum [24,21]. This argument shows that the universal approximation theorem can be obtained as the limit of a stochastic gradient based optimization procedure under an appropriate choice of hyper-parameters. In the scaling limit, our analysis gives bounds on the error of a representation and characterizes the asymptotic fluctuations in that error. Convergence to the optimum to first order occurs rapidly, i.e. on O(1) timescales. Diminishing the error at next order requires quenching the noise in the dynamics on O(log n) time scales.
Implications of noise in descent dynamics-Our results give an explicit theoretical explanation for the observation that additional noise in can lead to better generalization for neural networks [25,26]; local minima of depth O(n −1 ) are washed out by the noise of SGD.
Numerical experiments-We verify the scaling predicted by our asymptotic arguments for single layer neural networks. Because it is impossible to determine the exact interaction potential in general, we carry out numerical experiments using stochastic gradient descent for ReLU neural networks. We use the p-spin energy function [27,28] as the target function due to its complexity as the dimension grows large.
Key assumptions-In order to derive the stochastic partial differential equation for SGD, we effectively assume the large data limit. Because we are focusing on function approximation we can always generate new training data by sampling random points in the domain of the function and evaluating the target function at those points. The partial differential equation for gradient descent represents the evolution of the parameters on the true loss landscape, i.e., the large data limit. In this limit, the dynamics is similar to online algorithms for stochastic gradient descent [5].

In this manuscript the authors show that neural networks can be used to get approximations to given high-dimensional function with controllable errors. In particular, they consider the case where the function parameters satisfy either Gradient Descent or Stochastic Gradient Descent dynamics and prove law of large numbers and central limit theorems for the function the neural network outputs. They then use simulations, where the input function is a spherical 3-sping model, to show the performance of neural networks in estimating these functions.  I find the paper to be very clear and well-written. The authors do a good job in setting up the problem, discussing the two different settings and explaining the results. Those results are in my opinion very relevant since they shed new light, and provide important insights, in the strength of neural networks for representing complex high-dimensional functions, which can have several important applications in both theory and practice.  The authors have chosen to put all technical derivations and details in the supplementary material. Given the depth of the paper I understand this choice. However, the links between the parts of the main text and the supplementary material are not very clear. For instance, the authors use S1 in the main text to refer to Section 1 in the supplementary material, although sections there are simply numbered by 1, 2 and so on. Moreover, I found it hard to move between the main and supplementary parts in a continuous way. I ended up first reading the main text and then the supplementary material, which made it very hard to directly related the results in the latter to the main text. Here the problem arises that not everything used in the supplementary material is defined there, forcing the reader to go back to the main text and finding the missing definitions. Finally, I want to mention that the supplementary material does not follow the standard setup of first given a statement and then a detailed proof of it. Instead the results are preceded by lengthy computations which should represent the proof of this statement. This works in some case, but in most I was unable to parse the derivations I just read to get the proof of the result. I therefore stopped reviewing the computations after page 6 because of the difficulties in parsing them into needed proofs. To summarize, the paper investigates the very interesting and important usage of neural networks for representing complex high-dimensional functions using input parameters coming from given dynamics. The results seem solid and establishing the relation between function representation by neural networks and particle dynamics can potentially lead to better understanding and new advances of these systems. However, the way in which the results and technical derivations are split between the main text and supplementary material make it very difficult to properly check and verify the results. Still, I would recommend this paper for acceptance in NIPS, given that the authors take some time to better structure their results.  ADDED AFTER FEEDBACK: I remain convinced that the approach in the paper and results are interesting enough to validate acceptance in NIPS 2018. Although I strongly encourage the authors to restructure their supplementary material and discuss the contributions of the paper more clearly.  Below is a detail list of comments. Main text: Line 14, [batch size]: It is not clear to the reader at this point what this means. Line 31, [Progress analyzing…]: I would write: Progress on analyzing. Line 56: Another instance where batch size is used without having explained what is means. Line 59: I was confused by: justification explanation. It feels that it should be just one of these words. In addition, I would remove [in] in: noise in can lead… Line 70, [an exacting test]: Don’t you just mean: a test? Line 74-75: I would remove: is the following criterion, since it repeats the part of the sentence that precedes it.  Equations (4) and (5): Here the use of h is confusing since it was just use to denote an arbitrary test function. Line 86, [In this case, we…]: I would write: In this case, since we… Also, I would remove [and] in: and we can view… Line 92, [converges]: What type of convergence? I guess you mean pointwise.  Line 104, [synergies will…]: I would write: synergies which will… Line 114: As mentioned before, there is no Section S1. Moreover, it really is Section 1.1 in the supplementary material. I would recommend to be more specific here to avoid confusion and unnecessary text scanning by the reader. This holds for all subsequent occurrences of S references.  Equation (15): It is unclear with respect to what variables the integrals are taken. Please be explicit here. Equation (16): What is \rho_0? It is never defined and clearly cannot be a case of eq (13). Could you please give some more details here. Line 121: I would remove [and] in: point is and… Line 125: What is the difference between f_n(t) and f_n(x)? This is not clear from the text. Line 127: The \mathbb{P}_{in} comes out of nowhere which creates confusion. I would advise to spend some words on this before the statement of Proposition 3.1. Equation (19): Is this convergence uniform in t? I was not able to extract this from the proof and it feels that this is somewhat necessary to make the result applicable. This also applies to eq (29). Line 160: Here you refer to equation (27) which comes only later. I would recommend to follow the practice of not referring ahead and first give (27). In addition, I would write: (27) is statistically… Line 162, [spation-temporal]: I am not completely familiar with the terminology in Langevin dynamics but I would think that spatial-temporal is the right term. Line 172: What are \bm{\eta} and \eta? I understand that they refer to noise terms but it would be better to at least refer to the equation in the supplementary material that defines them. Line 192, [of noise term…]: I would write: of the noise term… Line 205: It would help the reader if you explain what S^{d-1}(\sqrt{d}) is. Figure 1: This figure is unclear. For instance, what do the different color stars mean? I would strongly recommend to add more explanation in the caption. Also, the figure is never explicitly referenced in the main text. Line 241, [that the]: I would replace this with: where the Supplementary material: Line 12: Are there really no restrictions on the input distribution \mathbb{P}_{in}? Line 14: What is \Omega? This is never defined. Line 37: What does the half arrow mean? I guess it means convergence in distribution. Please be more specific. Line 42-Equation (16): It is not clear why any of this is needed, since it does not seem to be used in the derivations that follow. Moreover, what is the derivative in eq (11)/(13)? Is this the Fréchet derivative? If so then why does \frac{\partial \mathcal{E}_0}{\partial \rho_0} seem to be a function, while \mathcal{E}_0 is a functional? It would really help if you give more details and a motivation for this part. Equation (18): I did not immediately see who (10) can be rewritten as (18). Could you please add some additional derivations? Line 58, [By the assumptions…]: Which assumptions are you referring to? The only assumption I could find here is 1.2 which says the kernel is continuously differentiable but I do not see why that is enough. Proposition 1.3: I was unable to transform the computations preceding this proposition into a proof of it. Please try and make the proof more apparent from the computations so that the reader does not need to solve this puzzle themselves. Line 72: I did not understand why the minimizers of \mathcal{E}_0 are relevant. Could you please elaborate a bit on this? Line 93: There seems to be a typo here. Along is written twice. Equation (32): I found the notation here quite confusing. First it is only revealed later that \xi(t) also depends on n which is indeed needed for else (32) would not be true in general. Second, what is the difference between \tilde{\rho}_\xi(t) and \tilde{\rho}_{\xi(t)}? Equation (38): What asymptotic are you considering here? Is it n \to \infty, t \to \infty or both n, t \to \infty?


Randomized Prior Functions for Deep Reinforcement Learning
Dealing with uncertainty is essential for e cient reinforcement learning. There is a growing literature on uncertainty estimation for deep learning from fixed datasets, but many of the most popular approaches are poorlysuited to sequential decision problems. Other methods, such as bootstrap sampling, have no mechanism for uncertainty that does not come from the observed data. We highlight why this can be a crucial shortcoming and propose a simple remedy through addition of a randomized untrainable 'prior' network to each ensemble member. We prove that this approach is e cient with linear representations, provide simple illustrations of its e cacy with nonlinear representations and show that this approach scales to large-scale problems far better than previous attempts.

Summary: This paper studies RL exploration based on uncertainty. First, they compare several previously published RL exploration methods and identifying their drawbacks (including illustrative toy experiments). Then, they extend a particular previous method, bootstrapped DQN [1] (which uses bootstrap uncertainty estimates), through the addition of random prior functions. This extension is motivated from Bayesian linear regression, and transferred to the case of deep non-linear neural networks. Experimental results on the Chain, CartPole swing-up and Montezuma Revenge show improved performance over a previous baseline, the bootstrapped DQN method.   Strength:  See conclusion.  Comments: - Lemma 2: The lack of uncertainty propagation in standard supervised learning methods is an important point indeed. However, you specifically introduce it as a problem of variational inference, while I think it is a problem that applies to all uncertainty methods from supervised learning used in RL, when their output distribution and loss only concern a mean. The problem indeed is that one needs to explicitly learn/propagate entire distributions or higher-order moments, i.e. specifying additional network outputs apart from the mean, as you mention in the appendix (lines 527-529). I think [5] is an early reference that identified this problem.  I don’t think the bootstrap method used in this paper is robust against this problem either actually. If I sample many 1-step bootstrap predictions for a particular state-action pair (and do sample the uncertainty at the next state), then for many samples the bootstrap datasets will become similar, and the entire ensemble starts to fit the same model (uncertainty becoming zero). This is because the loss in bootstrapped DQN still only considers/learns the mean of the distribution which you sample from.  - Figure 2: Why is the posterior over x^t*theta almost only varying in the x1 direction? The observed datapoints seem to have equal amount of spread in both the x1 and x2 dimensions. - Lemma 3: This is a key step in your method. I would prefer a derivation of both eq 4 and 5 in the appendix, showing how they lead to the posterior of eq 3.  - Line 196: At first it appeared strange to me that the random prior is fixed throughout learning, as for example many previous approaches resample parameter space noise [4] at the beginning of every episode. It might be good to identify that this choice does follow from Lemma 3, i.e. for each sample \tilde{theta} from the prior you get a sample from the posterior, and each bootstrap network covers one such sample from the posterior.  - Section 4.1: Are the prior networks also Glorot initialized (and then rescaled by beta)? Or what other distribution do you draw these from?  - Appendix B&C: (Chain results) I do not understand what parameter p denotes here. In Lemma 1 and Figure 12, it is the dropout probability. Line 608-609 mentions it as well? - Line 270: … and the continuous action space? - Figure 6: You seem to tune hyperparameters for your BSP, but not for the BS method? I think it would also be good to state how you compare to other directed exploration methods on Montezuma, e.g. [2,3] come to 3500 and 3000 score, respectively (which your best result is close to).  Appendix: - Lemma 5: I think this needs a proof/derivation of the lemma?  - Line 584: ‘where the data has target value zero, bootstrapping will always produce a zero target’ - - > Is this really true? The bootstrapping procedure is oblivious of the numeric range of the data right? I wonder whether the effect in Figure 9 would also be visible if your left 9 datapoints had y=1 instead of y=0. I think what you are looking at here is the extrapolation of a line when all measured datapoints nearby have the same y. It turns out that naive bootstrap methods then apparently have problems, but I don’t think this has to do with the fact that the y values are specifically 0. (This specific example is still relevant, as in sparse reward domains you will indeed see many returns of 0.) - Table 1: I think it would help to caption the column items, as some of these names were not explicitly mentioned in the text. For example, ‘learned metric’ at first confused me, but checking the crosses and reading back it probably refers to the discongriuity between new states and value uncertainty. -Table 1: ‘multi-step’ means whether uncertainty is propagated? I don’t think that is explicitly done for ensemble/bootstrap methods either right? Or do you mean that you sample your next state prediction Q(s’,a’) from the bootstrap ensemble distribution? But that does not prevent premature convergence of uncertainty, as the bootstrap distribution will still be fitting the mean.  - Appendix B: ‘Algorithm 14’ - - > numbering mistake?  Conclusion: At first, the proposed modification of random prior functions appeared simple, reminding of random parameter space noise. However, it is more involved and has a deeper theoretical motivation from Bayesian linear regression, and turns out to clearly work better than previous bootstrap ensemble methods in RL. Moreover, I really like the effort in Section 2 and Appendix A to systematically discuss previous attempts with respect to uncertainty in RL (although I think the lack of uncertainty propagation identified for varimethods is actually still present in the current paper). I think the paper could be clarified at a few points (as mentioned in the comments above), such as a derivation of Lemma 3 which is a key step in the paper, and the experimental section could mention results of other methods than the bootstrap DQN paper only. Overall, the paper contributes interesting insights and a novel, simple extension to the statistical bootstrap, which actually would be of interest to NIPS attendees outside RL (e.g., from Bayesian deep learning) as well.   [1] Osband, Ian, et al. "Deep exploration via bootstrapped DQN." Advances in neural information processing systems. 2016. [2] Bellemare, Marc, et al. "Unifying count-based exploration and intrinsic motivation." Advances in Neural Information Processing Systems. 2016. [3] Ostrovski, Georg, et al. "Count-Based Exploration with Neural Density Models." International Conference on Machine Learning. 2017. [4] Plappert, Matthias, et al. "Parameter space noise for exploration." arXiv preprint arXiv:1706.01905 (2017). [5] Dearden, Richard, Nir Friedman, and Stuart Russell. "Bayesian Q-learning." AAAI/IAAI. 1998.  After rebuttal: The authors elaborated on how their approach does propagate uncertainty, since each bootstrap network only uses itself as a target. I agree that this will propagate uncertainty. I think it is crucial to stress this aspect of their method, around Lemma 2 and probably somewhere in the introduction or discussion, as the authors indicate themselves in the rebuttal as well.


Embedding Logical Queries on Knowledge Graphs
Learning low-dimensional embeddings of knowledge graphs is a powerful approach used to predict unobserved or missing edges between entities. However, an open challenge in this area is developing techniques that can go beyond simple edge prediction and handle more complex logical queries, which might involve multiple unobserved edges, entities, and variables. For instance, given an incomplete biological knowledge graph, we might want to predict what drugs are likely to target proteins involved with both diseases X and Y?-a query that requires reasoning about all possible proteins that might interact with diseases X and Y. Here we introduce a framework to efficiently make predictions about conjunctive logical queries-a flexible but tractable subset of first-order logic-on incomplete knowledge graphs. In our approach, we embed graph nodes in a low-dimensional space and represent logical operators as learned geometric operations (e.g., translation, rotation) in this embedding space. By performing logical operations within a low-dimensional embedding space, our approach achieves a time complexity that is linear in the number of query variables, compared to the exponential complexity required by a naive enumeration-based approach. We demonstrate the utility of this framework in two application studies on real-world datasets with millions of relations: predicting logical relationships in a network of drug-gene-disease interactions and in a graph-based representation of social interactions derived from a popular web forum.

This work proposes a method for answering complex conjunctive queries against an incomplete Knowledge Base. The method relies on node embeddings, and trainable differentiable representations for intersection and neighbourhood expansion operations, which can compute a prediction in linear time relative to the edges involved. The method is evaluated on a biomedical and a reddit dataset. The originality of this work lies in posing "soft" logical queries against a "hard" graph, rather than posing "hard" queries against a "soft" graph, as is done typically.  The manuscript is clear, well-written and introduces notation using examples. The graphs and visualisations chosen throughout the manuscript are well-designed and help convey technical points. Careful thought has been put into selecting negatives during training, and there is an interesting ablation on different bilinear interaction functions.  A major criticism of this work is that it is not compared against standard KB inference methods which operate on the individual links in a query independently. Of course standard KB inference methods cannot be used directly to answer complex queries, but a standard query language can, and in the simplest form it could just aggregate scores of all facts involved, independently scored by a standard KB inference method. The authors mention that "this would require computation time that is exponential in the number of edges" (line 41-42). However, in this simplest approach, a model could compute edge scores independently, which is linear in the number of query edges. This would at least reveal whether it is important to score edges in conjunction (the implicit underlying premise of this work), or whether it suffices to treat them independently at test time.   A second criticism is that it is unclear whether the datasets chosen do actually test for a model's ability to answer logical conjunctions. 10% of edges are removed uniformly at random, and what exactly is required to answer a query is unclear. For example, in parallel query DAG structures, one branch alone might already suffice to unequivocally point to the correct answer. In fact, one observation from the experimental results is that linear chains are much harder to predict correctly than graphs with some sort of parallel structure. Furthermore, the longer the linear chain, the harder the task seems. This raises the question of redundancy in the dataset, and whether in those graphs with parallel structure one branch is already sufficient to specify the correct answer.     Questions / Comments to the authors: - notation: line 80-81: double use of tau? - notation (Theorem 1, third line): What is V?  - You mention different metrics: ROC AUC (section 5.1) and accuracy (Table 1) - training via sampling a fixed number of each query DAG structure was to facilitate batching? - line 272: where can this result be found? - is the macro-average across query DAG structures computed with adequate proportions, or uniformly weighted? - Can any general lessons be drawn from the selection of \Psi? - The origin of the drug interaction network dataset is not clear. How was this obtained?


Beyond Log-concavity: Provable Guarantees for Sampling Multi-modal Distributions using Simulated Tempering Langevin Monte Carlo
A key task in Bayesian machine learning is sampling from distributions that are only specified up to a partition function (i.e., constant of proportionality). One prevalent example of this is sampling posteriors in parametric distributions, such as latent-variable generative models. However sampling (even very approximately) can be #P-hard.
Classical results (going back to [BÉ85]) on sampling focus on log-concave distributions, and show a natural Markov chain called Langevin diffusion mixes in polynomial time. However, all log-concave distributions are uni-modal, while in practice it is very common for the distribution of interest to have multiple modes. In this case, Langevin diffusion suffers from torpid mixing.
We address this problem by combining Langevin diffusion with simulated tempering. The result is a Markov chain that mixes more rapidly by transitioning between different temperatures of the distribution. We analyze this Markov chain for a mixture of (strongly) log-concave distributions of the same shape. In particular, our technique applies to the canonical multi-modal distribution: a mixture of gaussians (of equal variance). Our algorithm efficiently samples from these distributions given only access to the gradient of the log-pdf. To the best of our knowledge, this is the first result that proves fast mixing for multimodal distributions in this setting.
For the analysis, we introduce novel techniques for proving spectral gaps based on decomposing the action of the generator of the diffusion. Previous approaches rely on decomposing the state space as a partition of sets, while our approach can be thought of as decomposing the stationary measure as a mixture of distributions (a "soft partition").
We address this problem by combining Langevin diffusion with simulated tempering. The result is a Markov chain that mixes more rapidly by transitioning between different temperatures of the distribution. We analyze this Markov chain for a mixture of (strongly) log-concave distributions of the same shape. In particular, our technique applies to the canonical multi-modal distribution: a mixture of gaussians (of equal variance). Our algorithm efficiently samples from these distributions given only access to the gradient of the log-pdf. To the best of our knowledge, this is the first result that proves fast mixing for multimodal distributions in this setting.
For the analysis, we introduce novel techniques for proving spectral gaps based on decomposing the action of the generator of the diffusion. Previous approaches rely on decomposing the state space as a partition of sets, while our approach can be thought of as decomposing the stationary measure as a mixture of distributions (a "soft partition").
Additional materials for the paper can be found at http://tiny.cc/glr17. Note that the proof and results have been improved and generalized from the precursor at http://www.arxiv.org/abs/1710.02736. See Section ?? for a comparison.

Summary:  This paper considers the problem of sampling from multi-modal distributions. Specifically they are interested in sampling from distributions that are (close to) mixtures of log-concave distributions with the same shape.   The algorithm that they consider for this task is a simulated tempering algorithm which utilizes Langevin dynamics Markov chains at varying temperatures. They show that this algorithm can produce samples close to the desired distribution in polynomial-time.   Quality:  The paper appears to be technically sound, although I have not verified the proofs rigorously.    Clarity:  This submission is easy enough to read, although it is light on details. Since one of the major tools/contributions of the paper is the decomposition theorem in Appendix D, I would have preferred if some statement of this theorem had appeared in the submission itself.   Originality:  This paper appears to present the first provably efficient algorithm for sampling from mixtures of log-concave distributions using only gradient information of the log-pdf.   Moreover, although decomposition theorems have previously been used to analyze Markov chains, the decomposition theorems presented in this paper appear to be novel as well.   Significance:  I think that this is a very interesting theoretical contribution to the study of sampling algorithms. Sampling from multi-modal distributions is an important problem in machine learning, and this paper makes progress in showing that a classical heuristic (simulated tempering) can be applied in this particular setting. Moreover, I think the techniques developed in this paper may be useful to others in this field.  That said, I do not know of any practical situations in which there is a distribution that we want to sample from that (a) is a mixture of log-concave distributions with the same shape and (b) we have only access to the gradient of its log-pdf. I think this submission would be made significantly stronger with a discussion of such situations.   Detailed comments:  - In the last sentence of the abstract, it states “…this is the first result that proves fast mixing for multimodal distributions.” However, since there are many Markov chains that provably sample from multimodal distributions, I think you probably meant something to the effect of “…with only access to the gradient of the log-pdf.”  - In lines 58-59 and 102-104, you discuss the lower bounds presented in Appendix K. I think it is important to clarify that you are talking about algorithms that have only access to the log-pdf and its gradient.


A Bandit Approach to Multiple Testing with False Discovery Control
We propose an adaptive sampling approach for multiple testing which aims to maximize statistical power while ensuring anytime false discovery control. We consider n distributions whose means are partitioned by whether they are below or equal to a baseline (nulls), versus above the baseline (actual positives). In addition, each distribution can be sequentially and repeatedly sampled. Inspired by the multi-armed bandit literature, we provide an algorithm that takes as few samples as possible to exceed a target true positive proportion (i.e. proportion of actual positives discovered) while giving anytime control of the false discovery proportion (nulls predicted as actual positives). Our sample complexity results match known information theoretic lower bounds and through simulations we show a substantial performance improvement over uniform sampling and an adaptive elimination style algorithm. Given the simplicity of the approach, and its sample efficiency, the method has promise for wide adoption in the biological sciences, clinical testing for drug discovery, and online A/B/n testing problems.

This paper considers adaptive allocation strategy that guarantees FWER or FDR in n distinct one-armed bandit problems (two-armed bandit problems with parameter \mu_0 known). While I think the results are interesting and the technologies used in this paper are sound (at least with some modifications), the current exposition of paper has many spaces for improvements. Mathematical notions are somewhat loosely used (detailed in major/minor remarks). The contribution of the proposed algorithm is not very easy to follow.  On the structure of the paper, (i) 1.3 Related work is rather long. In 8-page submission, a typical length of related work would be < 1 page.  (ii) 1.2 summary and 3 main results are overlapping: the first appearance of the main result should be presented with the main theorems.  Major remarks  >On Table 1  (i) To my understanding, these bounds are upper bounds. Is there any corresponding lower bounds?  (ii) What is the number of samples when uniform sampling is adopted? Are improvements of Alg 1 over uniform sampler significant? (iii) Use landau notation O(n \Delta^-2) for upper bound \Omega(n \Delta^-2) for lower bound.  >Algorithm 1 Alg 1 is not fully algorithmic because the lines begin with “Apply” are not stated algorithmically (appears to be detailed in Sec 2.1).  >On the novelty of Alg 1.  The algorithm utilizes standard multiple testing tools (BH and Bonferroni corrections) and the anytime p-value. One fear is that Alg 1 is a naive combination of these technologies. Could the authors highlight the innovative part of the Alg 1?  Minor remarks  Def 5: smallest time: Is sample complexity random variable?  Remark 1 takes me a significant amount of time to understand: I finally got the idea that no algorithm can stop with guarantee of TPR and FWPD, but I believe that the presentation can be improved. Line 97-98 hold only for one-sided test where \mu_i >= \mu_0. Line 133 p-value is undefined at this point Line 211/227 FDR(FWER)-controlled - these properties of sets are undefined. Line 262: independent of all problem parameters: universal constant? Line 265: do not use footnote right after math mode (c^3 looks like c-cubic). Line 267: Do not redefine H_1, H_0, Deltas. Line 271: Neither argument of the minimum follows from the other. > I could not understand.  #after author response The author rebuttal is helpful to me as they make some points clear (especially, the lower bounds: line 109-119).  The authors state that the sample complexity is a (high-probability-bounded) random variable, which sounds unconventional to me (for example, sample complexities H1 and H2 in "Best Arm Identification in Multi-Armed Bandits" are dist-dependent constants).   # Contribution and Presentation As stated in Appendix A, one of the novel contributions of this paper is to avoid union bound over arms to improve logarithmic factors, which I think novel. I found no critical error in the paper. I feel the paper is not very well written in representing their mathematical contents. Some formulas are redefined (e.g. H1, H0, Delta_i, Delta in Thm 1). Lemmas in Appendices are even more sloppy: For example, it takes some efforts to check whether or not P_{i,t} and S_t in pp.11-12 are the same as the ones defined in the main paper. Lemma 2 defines T_i that is used in S_t, which is not defined in the statement of Lemma 2.


Online Improper Learning with an Approximation Oracle ⇤
We study the following question: given an efficient approximation algorithm for an optimization problem, can we learn efficiently in the same setting? We give a formal affirmative answer to this question in the form of a reduction from online learning to offline approximate optimization using an efficient algorithm that guarantees near optimal regret. The algorithm is efficient in terms of the number of oracle calls to a given approximation oracle -it makes only logarithmically many such calls per iteration. This resolves an open question by Kalai and Vempala, and by Garber. Furthermore, our result applies to the more general improper learning problems.

This paper mainly considered the online linear optimization problem with approximation oracle (as well as its bandit version), which had been studied for a long time. The problem is important in real world especially when we can only solve the underlying offline optimization problems approximately, such as max-cut etc. The main contribution of this paper is to propose several algorithms which are proved to be more efficient (in terms of oracle calls per round) than previous state-of-the-art algorithm while achieve nearly the same regret guarantees. In detail, authors improved previous O(\sqrt{T}) oracle complexity per round to O(log T) order (both achieve O(\sqrt{T}) regret) in full information setting, and improved previous O(T) oracle complexity per round to O(T^{2/3}*log T) order (both achieve O(T^{2/3}) regret) in bandit setting. The paper is well written and structured in a clean way. Some techniques look interesting, especially the projection and decomposition oracle.    Pros: 1. Two algorithms based on different frameworks (i.e. OMD and continual exponential weights) are proposed in full information setting. Both of them are more efficient than previous methods. OMD based algorithm looks more succinct while only holds for loss functions with PNIP property. Continual exponential weights based algorithm works for general loss functions.  2.  An algorithm is designed for bandit setting based on above OMD type algorithm with a novel regularization function, which is also more efficient than previous algorithms.   Cons: 1. A critical condition assumed in this paper is that the mean of any log-concave distribution can be exactly calculated, and all of the proposed algorithms heavily rely on calculating the gravity center of underlying constraint set. However, in practical applications, we need to approximate this gravity center, and it will cause some approximation error, which will demonstrate a trade-off between computational efficiency and final regret performance and it is exactly my main concern about this paper. According to the explanation in appendix D, this approximation error \epsilon is set as 1/T, then calculating the approximate gravity center will cost time poly(1/\epsilon) = poly(T) per round, which degenerates the computational performance of proposed algorithms. If we want to guarantee the efficiency of approximate mean calculation algorithm, then \epsilon should be set at least in 1/\log(T) order, then I am not sure whether this approximation error is enough to guarantee the final O(\sqrt{T}) regret performance; 2. Could authors explain why OMD based algorithm can achieve better efficiency than the algorithm proposed in Garber 2017 paper? I know his algorithm is based on Online Gradient Descent and ellipsoid method. Maybe the critical part is the projection and decomposition oracle constructed in this paper. It will be better if authors could provide some intuition about this oracle before stating four detailed lemmas.  Other minor comments: 1. In table 1, it will be better to add a notation about alg 6, as it only appears in the appendix; 2. In the equation under line 307 in appendix, it should be f(x) – f(y) <= \frac{1}{2\mu} …  Update: Thanks for authors' response. Though the main focus of this paper is oracle complexity, but I think computational complexity is also a critical part. Since authors mentioned using ellipsoid method could lead to a better computational complexity, which solved my main concern, I hope to see the corresponding part in the further version.


Distributed k-Clustering for Data with Heavy Noise
In this paper, we consider the k-center/median/means clustering with outliers problems (or the (k, z)-center/median/means problems) in the distributed setting. Most previous distributed algorithms have their communication costs linearly depending on z, the number of outliers. Recently Guha et al. [10] overcame this dependence issue by considering bi-criteria approximation algorithms that output solutions with 2z outliers. For the case where z is large, the extra z outliers discarded by the algorithms might be too large, considering that the data gathering process might be costly. In this paper, we improve the number of outliers to the best possible (1 + )z, while maintaining the O(1)-approximation ratio and independence of communication cost on z. The problems we consider include the (k, z)-center problem, and (k, z)-median/means problems in Euclidean metrics. Implementation of the our algorithm for (k, z)-center shows that it outperforms many previous algorithms, both in terms of the communication cost and quality of the output solution.

The paper describes algorithms for distributed k-center/k-median/k-means clustering. For distributed k-center, it is known that any constant approximation needs Omega(z) communication cost, where z is the number of allowed outliers. This can be avoided by using a bicriteria approximation where more than z outliers are allowed, but the cost is compared to the best solution with at most z outliers. It is known that using 2z outliers is sufficient to obtain a O(1)-approximation. The paper at hand now improves the violation with respect to the number of outliers to (1+eps)*z. The result is given for metric k-center, but works for Euclidean k-center as well (since Euclidean distances are a metric, and only O(1) is obtained anyway).  For Euclidean k-median/k-means, a similar result is obtained using coresets. However, here the interpretation is a bit different, since these problems can (under no assumptions on n and d) not be (1+eps)-approximated. (This still needs to be done by the coordinator after collecting the information). Nevertheless, coresets have proven useful for plain k-median/k-means as well, so I consider this extension to be meaningful.  The paper is well written and contains a helpful figure for understanding the main proof. The algorithm is elegant.   The experiments have not made it into the actual paper, which I think is not so good. The experiments do look solid and compare several approaches. I hope that a long version of this paper is going to be published soon after the publication of the conference version so that this part of the paper is not lost to the reader. The one-sentence summary of it in the actual conference version is not very informative.  The overall accessment probably hinges on the question how important the improvement from 2z to (1+eps)z is.   - The sentence on [5] in line 114 should have k-median in it and not k-center. - To the conclusion: It seems unlikely that the problem can be solved within a 1+eps factor on the coreset for k-median/k-means, since the problems are even APX-hard without the outlier part.  Edit after rebuttal: Since I cannot pinpoint an exact minus point for this paper and the result is interesting, I've update my score to accept.


Deepcode: Feedback Codes via Deep Learning
The design of codes for communicating reliably over a statistically well defined channel is an important endeavor involving deep mathematical research and wideranging practical applications. In this work, we present the first family of codes obtained via deep learning, which significantly beats state-of-the-art codes designed over several decades of research. The communication channel under consideration is the Gaussian noise channel with feedback, whose study was initiated by Shannon; feedback is known theoretically to improve reliability of communication, but no practical codes that do so have ever been successfully constructed. We break this logjam by integrating information theoretic insights harmoniously with recurrent-neural-network based encoders and decoders to create novel codes that outperform known codes by 3 orders of magnitude in reliability. We also demonstrate several desirable properties in the codes: (a) generalization to larger block lengths; (b) composability with known codes; (c) adaptation to practical constraints. This result also presents broader ramifications to coding theory: even when the channel has a clear mathematical model, deep learning methodologies, when combined with channel-specific information-theoretic insights, can potentially beat state-of-the-art codes, constructed over decades of mathematical research.

I acknowledge that I read the author's response, which address many of my concerns. As a result, I am raising my score to a 5 -- I lean toward rejection but not strongly. I remain concerned about the absence of a related work section -- and the related work cited in the response still seems inadequate. In our discussions, R2 pointed out a lot of additional work the authors appear to have missed.  -----  Summary:  This paper describes a deep learning approach to encoding and decoding in a noisy channel with feedback, something that has been little explored in the literature to date. The formal noisy channel setting is similar to a standard autoencoder framework, with a few key differences. For one, we usually encode and transmit one bit of a message at time due to channel limits, and second we get feedback, usually in the form of a noisy version of each encoded bit. Due to the sequential nature of the problem, plus the availability of feedback, the authors apply an RNN architecture. The input to the decoder at each step is the next bit to encode plus an estimate of the noise from previous steps (derived from the difference between the encoded message and the received feedback). Experiments suggest that this approach significantly outperforms existing approaches. The authors explore several variants of the RNN architecture specific to the setting: normalization to restrict the power of the encoded message, per-bit weights to adapt the power of each encoded message (similar to an attention mechanism), and an encoder-decoder framework for the feedback.  On the one hand, I think this paper is pretty solid and could be accepted on the merits of its work.  On the other hand, the paper lacks entirely a related work section and fails to discuss ANY previous work on deep learning applied to noisy channels. A quick Google search indicates there is plenty of such work (as does the mysterious presence of uncited papers in the references, see below).  It is hard to imagine accepting the paper in its current form. I would like to hear the authors' explanation and discuss with the other reviewers before making a final decision. For now I will put a temporary "clear reject" score but am willing to revise it.  Quality:  Although a straightforward application for the most part, the work nonetheless appears high quality to me. The choice to apply an RNN seems sound and the various proposed architectures seem well-matched to the problem. The authors demonstrate a thorough understanding of the problem setting (though their description could be improved, see below). The results appear to speak for themselves: this approach soundly beats the baselines.  I especially appreciate Sections 4 and 5 -- the experiments in this paper are thorough, and the authors make a good faith effort to provide insights about their proposed model's behavior, rather than just reporting quantitative performance.  A few questions about the proposed approach and experiments:  - The authors need to justify the choice to use an RNN without first experimenting with a non-recurrent architecture. Do we really need to capture longterm dependencies? The coupling results (lines 313-318) only cover a very short history: we could easily design an MLP or convolutional architecture that accepts multiple steps of input to capture such history. Did the authors try this?  - The discussion of complexity (lines 234-243) is somewhat vague. I expected to see an actual complexity analysis (in terms of block length and maybe coded message length?) and then maybe some empirical timing experiments -- along with a comparison to existing algorithms (in both formal analysis and in the experiments). Instead we got a vague handwave suggesting that they're comparable. That's not very convincing.  Clarity:  This paper is for the most part well-organized and written (modulo the missing related work), but it may be difficult for readers who are unfamiliar with coding theory. To make it more accessible to a wider reader audience (thus increasing its potential impact), I suggest the following:  - clearly articulate how this problem differs from, say, standard autoencoding problems or machine translation, which both have similar looking structure.  - show the full end-to-end architecture with encoder AND decoder and explain how it works. It's still not clear to me whether the decoder is run once at the end, after the full coded message is transmitted (as implied by the bidirectional RNN architecture) or whether each decoding step happens after the corresponding encoded step is transmitted.  Some nitpicks:  - The justification for the first phase is hard to follow. Why do we want to send the un-encoded bits first? What would happen if we omitted that phase (maybe some results for an RNN with phase 1)?  - Why does the RNN in Figure 2 output two coded bits per step? Is that simply due to the rate of 1/3 per the equation in lines 164-165 -- so that with a different rate, we'd output more codes?  - Why is the input to the RNN at each step the difference between the feedback and codes, rather than just the raw feedback? Did the authors try this?  - What is meant by "unrolling" the RNN cells when working with longer block lengths (line 284)? Does this mean using truncated backprop through time or what?  - The description of the architecture used (lines 239-241) is a bit confusing: the authors say they used an RNN in the encoder but a GRU in the decoder. Is the encoder in fact a vanilla RNN or is this a typo?  Originality:  It is impossible to judge the paper's originality in the absence of a thorough discussion of related work. Why is there no related work section, or at least a discussion of relevant previous work in the introduction? This is especially baffling given that the references section lists multiple relevant papers [10-17] that are not cited anywhere in the paper. This means they're listed in the compiled bibtex database file, which in turn means that at some point, a version of the paper citing them must have been compiled. Was the related work section accidentally commented out? On the other hand, the paper is exactly 8 pages long, meaning that there's no room for related work anyway.  In any case, the paper's language, the lack of discussion of similar work, and the absence of other neural net baselines, implies this work may be the first to apply RNNs to the noisy channel with feedback problem. I am not sufficiently familiar with work in this area to confirm or deny that claim, but I am dubious of any claim to be the first to apply deep learning to a classic problem -- and the works listed in the references in this paper [10-17] (plus those turned up in a quick search) cast some doubt.  The authors need to tighten their claim: what exactly is their original contribution? And they need to position their work with respect to related research. If there are alternative approaches to this problem -- either previously published or even not published but obvious -- then they need to make a best effort to compare them rhetorically and empirically.  Significance:  The significance of an applied paper of this type is tied almost entirely to its empirical results and what new insights we can derive from them. Taking the paper at face value, it appears quite significant -- the proposed RNN architecture is substantially better than standard approaches, and I -- for one -- did not know that, even if it seems unsurprising.  However, given the omission of a related work discussion, it is impossible to fully decide the significance of this paper. If someone else previously applied RNNs to this problem and produced similar results, then this paper's contribution is diminished.


NEON2: Finding Local Minima via First-Order Oracles
We propose a reduction for non-convex optimization that can (1) turn an stationary-point finding algorithm into an local-minimum finding one, and (2) replace the Hessian-vector product computations with only gradient computations. It works both in the stochastic and the deterministic settings, without hurting the algorithm's performance. As applications, our reduction turns Natasha2 into a first-order method without hurting its theoretical performance. It also converts SGD, GD, SCSG, and SVRG into algorithms finding approximate local minima, outperforming some best known results.

This paper provides a way to turn algorithms that find a first-order stationary point (i.e. gradient norm is bounded by a constant) to algorithms that find a second-order stationary point (a.k.a. approximate local minima). The definition of a second order stationary point is that the gradient norm at the point is bounded by a constant AND the smallest eigenvalue of the Hessian is at least a negative number (but with small magnitude).  The main contribution here is that the proposed approach avoids Hessian or Hessian-vector product computations. It just needs (stochastic) gradient information, which is a significant contribution. Previous works need Hessian-vector products for computing the eigenvector that corresponds to the most negative eigenvalue of the Hessian; it is recognized that the negative curvature can be exploited to decrease the objective value even when the gradient norm is small.  In this paper the authors provide two methods to replace the Hessian-vector product computations. One for the offline setting, which is that setting that the convergence rate depends on number of samples n. The other for the online setting, which is the setting that the convergence rate does not depend on $n$, as $n$ can be infinite. Theoretical guarantees on both offline setting and online setting are provided.  I enjoy reading the paper. The analysis looks correct and sound, though I didn't have time to read them in details. One concern is that the complexity result in this paper may not be the state-of-the-art. "SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimator" might have a better rate. Yet, I understand that the paper is put on arxiv after this submission. It is unfair to compare them. Still, I suggest the authors to briefly mention it in the next version.  (p.s. It seems like SPIDER has a state-of-the-art rate in finding a first order stationary point. The authors of SPIDER follow the nice tricks proposed in this paper to exploit the negative curvature and turn it to an approximate local-minimum finding one. So, the papers are actually complementary.)


Inferring Networks From Random Walk-Based Node Similarities
Digital presence in the world of online social media entails significant privacy risks [31,56]. In this work we consider a privacy threat to a social network in which an attacker has access to a subset of random walk-based node similarities, such as effective resistances (i.e., commute times) or personalized PageRank scores. Using these similarities, the attacker seeks to infer as much information as possible about the network, including unknown pairwise node similarities and edges. For the effective resistance metric, we show that with just a small subset of measurements, one can learn a large fraction of edges in a social network. We also show that it is possible to learn a graph which accurately matches the underlying network on all other effective resistances. This second observation is interesting from a data mining perspective, since it can be expensive to compute all effective resistances or other random walk-based similarities. As an alternative, our graphs learned from just a subset of effective resistances can be used as surrogates in a range of applications that use effective resistances to probe graph structure, including for graph clustering, node centrality evaluation, and anomaly detection. We obtain our results by formalizing the graph learning objective mathematically, using two optimization problems. One formulation is convex and can be solved provably in polynomial time. The other is not, but we solve it efficiently with projected gradient and coordinate descent. We demonstrate the effectiveness of these methods on a number of social networks obtained from Facebook. We also discuss how our methods can be generalized to other random walk-based similarities, such as personalized PageRank scores. Our code is available at https://github.com/cnmusco/graph-similarity-learning.
In this work we focus on these random walk-based similarity metrics. We initiate the study of a fundamental question:
How much information about a network can be learned given access to a subset of potentially noisy estimates of pairwise node similarities?
This question is important from a privacy perspective. A common privacy breach is social link disclosure [6,56], in which an attacker attempts to learn potentially sensitive links between nodes in a network. Such attacks are very common; fake accounts are used to infiltrate social groups, potential employers may want to inspect a job candidate's social network, and advertisers may wish to probe a user's information to offer targeted ads. Thus, studying the ability of an attacker to reveal link information using node similarities is important in understanding the privacy implications of releasing similarities, or information that can be used to compute them.
There are many scenarios in which node similarities may be released, either directly or indirectly, with the potential to reveal private link information. For example, when searching for users on a social network platform, node similarity is indirectly revealed since similar users (in terms of social connections) are often displayed together in search results. As a second example, random walk-based graph embeddings (e.g. PageRank or DeepWalk embeddings) may be released publicly for research purposes since, naively, they appear to contain no identifying information.
From a data mining perspective, computing all pairwise node similarities can be infeasible for large networks since the number of similarities grows quadratically in the number of nodes. Additionally, when the network cannot be accessed in full but can only be probed via crawling [28], we may only have access to similarity estimates rather than their exact values. Thus, understanding what information can be learned from a partial, potentially noisy, set of node similarities is important when using these metrics in large scale graph mining.
Finally, in some scenarios, it may be possible to measure node similarities for an underlying graph, which we cannot directly access but wish to recover. For example, in evolutionary ecology, effective resistance distances in planar "environment graphs" have been shown to correlate with genetic differentiation in geographically distributed populations [36,37,41]. In this context, measurements of geographic genetic variation give incomplete and noisy measurements of effective resistances. Recovering an underlying graph from these measurements corresponds to recovering plausible limitations on migration and movement that could have led to the observed genetic variations.

Updates after reading author response and other reviews  Thanks for the clarity on the egonets. I agree that perhaps the social structure can be interesting and this could be a strength rather than weakness.  I didn't actually have a concern on practical use cases, as the response states. I just think it would be more interesting to consider noise that appears from the types of situations described, rather than Gaussian (and the authors acknowledged this point).  Overall, this was an interesting submission. I am upping my score to 7.  --------------------  This paper proposes to learn an unknown graph given a subset of the effective resistances of pairs of nodes. The proposed methods can substantially out-perform random guessing in network reconstruction on a couple synthetic networks and some Facebook egonets.  The main idea of the paper is interesting and original, and the proposed methods do well on the datasets considered. However, I think the choice of egonets as real-world datasets is limiting. The fact that all nodes share a common neighbor implies substantial social structure (and possibly substantial graph structure, if the ego is included --- it is not clear if the ego is included in these datasets). I would suggest to instead use a variety of social networks, or if the authors are keen on Facebook, use some of the Facebook100 graphs.  Another weakness in the experimental design is the following. The noise terms for the experiments are Gaussian, but I read the motivation for the noise terms to come from using approximation algorithms to deal with computational constraints or from partial measurements (last paragraph of introduction, lines 43--48). The usefulness of the Gaussian error seems to be that the solution to the optimization problem (Problem 2) is the MLE. The experimental results would be much more convincing if the noise was closer to the motivation. For example, the authors could compute approximations to the effective resistance with an approximation algorithm (the simplest case might be to run just use a few steps of an iterative solver for computing the effective resistances).  From the presentation of the introduction, I expected more theoretical results. However, the only formal theoretical statements I see in the main text are Theorem 1, which was proved in another paper, and Proposition 1, which is a gradient calculation. Overall, I think this paper would greatly benefit by tightening up all of the language around the problem formulation and including more experiments and description of the actual methodology (e.g., what is the SDP formulation?).  All of that being said, I enjoyed the main idea of the paper. The problem itself opens up some interesting avenues for both theoretical and empirical future work, and the numerical experiments do show that the proposed methods can be successful. I think the experiments (namely, choice of real-world datasets and types of noise) and presentation could be updated for an improved camera version. Thus, I lean towards accepting this paper.  Fianlly, perhaps I am missing a compelling reason that we should be using egonets as the main example, so please point this out to me if it is the case.   Some small issues:  1. In the explanation of privacy attacks (lines 36--42), it would be useful to know some scenarios under which the node similarities would be released, but the link information would not be released.  2. "we show that, when G is a tree, exact recovery is possible for Problem 1.2 when S is a superset of G's edges" Should this be subset of edges?  3. Please include a reference to [43] in the statement of Theorem 1. Right now, it seems like the theorem is a novel contribution of this paper, which, from my reading of the text, it is not.  4. "This approach quickly converges to near global minimums for many networks." I am not sure how I am supposed to draw this conclusion from the experiments.  5. "This modified problem is convex and can be solved via an SDP." Where is the SDP formulation?  6. Why is there no \sigma^2 = 0.1 case for FB Medium A and FB Large A.  7. Typos: Appendixx E (line 186), we learn via share (line 236)


Synaptic Strength For Convolutional Neural Network
Convolutional Neural Networks(CNNs) are both computation and memory intensive which hindered their deployment in mobile devices. Inspired by the relevant concept in neural science literature, we propose Synaptic Pruning: a data-driven method to prune connections between input and output feature maps with a newly proposed class of parameters called Synaptic Strength. Synaptic Strength is designed to capture the importance of a connection based on the amount of information it transports. Experiment results show the effectiveness of our approach. On CIFAR-10, we prune connections for various CNN models with up to 96% , which results in significant size reduction and computation saving. Further evaluation on ImageNet demonstrates that synaptic pruning is able to discover efficient models which is competitive to state-of-the-art compact CNNs such as MobileNet-V2 and NasNet-Mobile. Our contribution is summarized as following: (1) We introduce Synaptic Strength, a new class of parameters for CNNs to indicate the importance of each connections. (2) Our approach can prune various CNNs with high compression without compromising accuracy. (3) Further investigation shows, the proposed Synaptic Strength is a better indicator for kernel pruning compared with the previous approach in both empirical result and theoretical analysis.

Content:   This submission introduces a new framework for compressing neural networks. The main concept is to define the "synaptic strength" of the connection between an input layer and an output feature to the the product of the norms of the kernel of the input layer and the norm of the input layer. A large synaptic strength indicates that a certain input feature plays a substantial role in computing the output feature.  The synaptic strength is incorporated into the training procedure by the means of an additional penalty term that encourages a sparse distribution of synaptic strengths. After training all connections with synaptic strength smaller than some threshold are fixed to zero and the network is finetuned.  The authors apply their method to different networks on CIFAR (including VGG, ReseNet and DenseNet) and ImageNet (ResNet-50) and show good performance (in terms of accuracy and number of remaining parameters and computations). They also look into how the sparsity after training affects accuracy.   Quality:  The idea of synaptic strength appears intriguing. I like especially the idea of pruning whole connections between input channels and output layers instead of individual weights and it's very interesing to see in which layers most pruning is happening (Figure 3 left).  I have some minor issues with the derivations in section 3.2: * eq (1): shouln't the bias be b^{i+1}_k instead of b_{i+1}_c, i.e. the each output feature (k) has a certain bias, not the input channels (c). The same should be the case for the final summand b_c in eq. (3)-(5). * In eq (3)-(5): the convolution and adding of the bias should happen inside the nonlinearity f, not after applying the nonlinearity. Maybe there are just some parantheses missing? * going from eq (3) to eq (4), the factor gamma is moved from inside the nonlinearity outside of it. This works for nonlinearities like relu and prelu (as long as gamma is positive!), but it won't work for other nonlinearities like sigmoids. This condition should be clarified.  The results reported in the paper show that the method manages to decrease the number of parameters while keeping good performance. One other quite recent method that I would like to see compared to in this paper is Louizos et al, "Learning Sparse Neural Networks through L_0 Regularization", ICLR 2018 which reports a performance of less than 4% on CIFAR 10 with WideResNets (compared to 5% in this submission with ResNet).  One of the reasons why I would like to see this comparison is that Loizos et al report that besides compressing the network, their method also serves as a regularization method that can improve generalization and I'm wondering whether the same might hold for the proposed method in this submission. Table 1 suggests that might be the case for VGG and ResNet-18 on CIFAR10 but not on ImageNet (Table 2).  Also I'm wondering how strong the dependence on the regularization parameter lambda is. How where the used parameters choosen and how much do the results differ if a different parameter is used?  The bold numbers in Table 3 indicate that the ResNet-50 version of the submission has with 5.9M parameters the smallest number of remaining parameters, however, 3 out of the 4 comparison models in the table have even less parameters.  What exactly is the sparsity in Figure 4 (a) and (b)? The parameter lambda? The threshold tau? Something else? Why did the authors not finetune in the ablation study? If the difference would go away after finetuning, this would change the result of the ablation study, right?   Clarity:  There are some details missing: What was the threshold for pruning connections?  The captions of tables 2 and 4 are not indicating which dataset they are using (ImageNet if I'm not mistaken).  Please have your submission proof-read for English style and grammar issues.   Originality:  The authors make an interesting contribution to the wide field of compressing neural networks. For me the main contribution is to make the connection between a single input feature and a single output feature the target of the compression. I'm not aware that this has been done before -- but my knowledge of this field is like not complete.  The authors compare to some other compression techniques but are missing on some recent state-of-the-art methods that show really good performance (see above for one example)  Significance:  Compressing DNNs is a very important topic for bringing deep learning into everyday applications. The concept of synaptic strength is an very interesting contribution to that field. To help convincing other researchers of this method, I would like to see more comparison with state-of-the art and other potential applications of the method like regularization.  Minor comments: In the caption of Figure 1, "left" and "right" are switched.   Update after the rebuttal:  The authors addressed most of my questions and concerns. With the promised changes incorporated I think the paper will be a better contribution and therefore I am happy to increase my rating from 6 to 7.