Practical Methods for Graph Two-Sample Testing
Hypothesis testing for graphs has been an important tool in applied research fields for more than two decades, and still remains a challenging problem as one often needs to draw inference from few replicates of large graphs. Recent studies in statistics and learning theory have provided some theoretical insights about such high-dimensional graph testing problems, but the practicality of the developed theoretical methods remains an open question. In this paper, we consider the problem of two-sample testing of large graphs. We demonstrate the practical merits and limitations of existing theoretical tests and their bootstrapped variants. We also propose two new tests based on asymptotic distributions. We show that these tests are computationally less expensive and, in some cases, more reliable than the existing methods.

This paper studies the problem of two-sample testing of large graphs under the inhomogeneous Erdos Renyi model. This model is pretty generic, and assumes that an undirected edge (ij) is in the graph with probability P_{ij} independently of all other edges. Most generically the parameter matrix P could be anything symmetric (zero diagonal), but common models are stochastic block model or mixed membership stochastic block model, which both result in P being low rank. Suppose there were two random graph distributions, parameterized by matrices P and Q, and the goal is to test whether P = Q or not (the null hypothesis being that they are equal).              They assume that the graphs are vertex-aligned, which helps as it reduces the problem of searching over permutations to align the graphs. If one has many samples from each distribution, and the number of samples goes to infinity, then there are kernel based tests constructed based on comparing the graph adjacency or Laplacian matrices. This paper focuses on the setting where there are only a few samples from each distribution, or even possibly only one sample from each distribution.               When the number of samples per distribution is greater than 1, then the paper uses test statistics previously introduced by Ghoshdastidar, one which looks at the spectral norm of differences between the mean adjacency matrix of the two distributions, and the other which looks at correlation of edges in the average adjacency matrices of the two distributions. The limitation of previous tests was that the theoretical thresholds used in the test are too large (for moderate graph sizes), so they estimate threshold by bootstrapping, which can be quite expensive. The contribution of this paper is to provide an asymptotic analysis of these test statistics as the number of vertices goes to infinity, showing that the test statistic is asymptotically dominated by a standard normal random variable. This suggests an easy to compute threshold to use for the hypothesis test which does not require bootstrapping samples to compute. The asymptotics are as the number of vertices goes to infinity rather than the number of samples going to infinity, but the separation condition between P and Q that the test can distinguish between decreases as the number of samples per distribution increases.               When there is only one sample per distribution, most previous methods are limited to assuming that P and Q are low rank. The authors propose a test based on asymptotic distribution of eigenvalues of a matrix related to the difference between adjacency matrices (but each entry scaled by a term related to the variances of that edge). Basically they show that if one knew the parameter matrices P and Q, then the extreme eigenvalues of matrix C defined in eq (9) follow the Tracy-Widom law, thus the spectral radius of C is a useful test statistic. Since in practice P and Q are not known, the authors then propose to estimate P and Q using community detection algorithms such as spectral clustering, where P and Q are approximated by stochastic block models. I think this last part is more heuristic as the authors did not mention whether the theoretical result in spectral clustering are tight enough to provide analogous results as Theorem 4 in the setting where P and Q are estimated.              I enjoyed reading the paper and found the results quite interesting and clearly presented.              Comment about organization:       I feel that since the formal problem statement pertains to the entire paper (m > 1 and m=1), it shouldn't be a subsection within the m \to \infty section; maybe make the problem statement its own section.


Quadrature-based features for kernel approximation
We consider the problem of improving kernel approximation via randomized feature maps. These maps arise as Monte Carlo approximation to integral representations of kernel functions and scale up kernel methods for larger datasets. Based on an efficient numerical integration technique, we propose a unifying approach that reinterprets the previous random features methods and extends to better estimates of the kernel approximation. We derive the convergence behaviour and conduct an extensive empirical study that supports our hypothesis 1 .

Post rebuttal: Thanks for the feedback.  Summary of the paper: This paper proposes a novel class of random feature maps that subsumes the standard random Fourier features by Rahimi and orthogonal random features by [16]. The approach is based on spherical radial quadrature rules by Genz and Monahan [18]. Some theoretical guarantees are provided, and impressive experimental results are reported.  ~~~ Quality ~~~ Strength: Extensive experimental results have been conducted, showing impressive empirical performance of the proposed approach. Also some theoretical guarantees are provided. Weakness: Theoretical error-bounds seem not very tight, and it is theoretically not very clear why the proposed works well; see the detailed comments below. I guess this would be because of the property of spherical radial rules being exact for polynomials up to a certain order.  ~~~ Clarity ~~~ Strength: This paper is basically written well, and connections to the existing approaches are discussed concisely. Weakness: It would be possible to explain theoretical properties of spherical quadrature rules more explicitly, and this would help understand the behaviour of the proposed method.  ~~~ Originality ~~~ Strength: I believe the use of spherical radial quadrature is really novel. Weakness: It seems that theoretical arguments basically follow those of [32].  ~~~ Significance ~~~ Strength: The proposed framework subsumes the standard random Fourier features by Rahimi and orthogonal random features by [16]; this is very nice, as it provides a unifying framework for existing approaches to random feature maps. Also the experimental results are impressive (in particular for approximating kernel matrices). The proposed framework would indicate a new research direction, encouraging the use of classical quadrature rules in the numerical analysis literature.    %%% Detailed comments %%%  ++ On the properties of spherical radial rules ++ You mentioned that the weights of a spherical radial rule [18] are derived so that they are exact for polynomials up to a certain order and unbiased for other functions. We have the following comments: - Given that [18] is a rather minor work (guessing from the number of citations; I was not aware of this work), it would be beneficial for the reader to state this property in the form of Theorem or something similar, at least in Appendix. From the current presentation it is not very clear what exactly the this property means mathematically. - Are spherical radial rules related to Gauss-Hermite quadrature, which are (deterministic) quadrature rules that are exact for polynomials up to a certain order and deals with Gaussian measures?  ++ Line 109 and Footnote 3 ++ From what you mentioned that with some probability $a_0$ can be an imaginary number. What should one do if this is the case?  ++ Proposition 3.1 ++ The upper bound is polynomial in $\epsilon$, which seems quite loose, compared to the exponential bound derived in the following paper: Optimal rates for random Fourier features  B. K. Sriperumbudur and Z. Szabo  Neural Information Processing Systems, 2015  ++ Proposition 3.2 ++ What is $\lambda$? Where is this defined?


Geometry-Aware Recurrent Neural Networks for Active Visual Recognition
We present recurrent geometry-aware neural networks that integrate visual information across multiple views of a scene into 3D latent feature tensors, while maintaining an one-to-one mapping between 3D physical locations in the world scene and latent feature locations. Object detection, object segmentation, and 3D reconstruction is then carried out directly using the constructed 3D feature memory, as opposed to any of the input 2D images. The proposed models are equipped with differentiable egomotion-aware feature warping and (learned) depth-aware unprojection operations to achieve geometrically consistent mapping between the features in the input frame and the constructed latent model of the scene. We empirically show the proposed model generalizes much better than geometryunaware LSTM/GRU networks, especially under the presence of multiple objects and cross-object occlusions. Combined with active view selection policies, our model learns to select informative viewpoints to integrate information from by "undoing" cross-object occlusions, seamlessly combining geometry with learning from experience.

The paper proposes a method for reconstructing, segmenting and recognizing object instances in cluttered scenes in an active vision context, i.e. in situations where the camera can be moved actively by an agent, for instance a robot. Active vision is investigated actively in the recent past, and there seems to be a convergence between different communities, which start to develop similar methodologies for different objectives: (i) research in agent control, in particular with POMDP/Deep-RL, tackles realistic settings requiring visual recognition and includes geometry in its research, and (ii) research in vision/object recognition discovered Deep-RL to tackle active vision and creates neural/differentiable formulations of geometry based methods.  The proposed method belongs to the second category and solves the problem with a 3D tensor representation, to which different subsequent views are unprojected and integrated. Depth and FG masks are estimated directly from RGB input using U-nets, and the different 3D representations are aligned with the first viewpoint in the sequence using ground truth odometry (using groundtruth ego-motion is standard in the community when working with simulated environments). From the 3D representation, which is updated using a recurrent layer, different output layers estimate different desired quantities: reconstruction map (voxel occupancy), segmentation map, etc. A policy output estimates the next view from a discrete action space.  The method is very interesting and addresses many open problems, in particular how to perform these estimations from cluttered scenes on multiple objects jointly.  Methodology-wise, it builds on existing work adding several key-components, in particular the 3D voxel-like representation which is updated with recurrent networks, as well as instance-wise output through clustering, which is responsible for the multiple-object capabilities.  Here, we can remark that the number of objects in the scene needs to be known beforehand, since clustering is performed using k-means instead of a more sophisticated method capable of estimating the amount of instances. This is ok for the performed experiments on the ShapeNet dataset but seems to be an important restriction, unless some of the discovered instances can be estimated as no-objects using an additional confidence output (like in the 2D object detection literature like YOLO, SSD etc.).  The paper is easy to read and to understand, and in large parts very easily understandable. However, that said, I have a couple of issues with the presentation and the lack of description of key details, which unfortunately concern the key differences to existing work. The paper is very dense and it is clearly difficult to present all the details, but the choices made here are not optimal.  The first (and minor) part concerns the recurrent memory update, which is intuitively clear, but which lacks equations which related all key quantities to each other. The updates are described of being “recurrent”, but here is necessary to provide clearer indications on what is input, what is output, is there a dedicated hidden recurrent layer or does the 3D tensor play the role of the recurrent representation etc. It is, of course, not necessary to provide all the detailed equations of the different gates of a complex GRU cell; however, an RNN-like description is compact and can provide all relevant details, while it can simply be mentioned that GRU gates are part of the implementation.  The second part concerns the way how the output is dealt with to estimate object instances. Firstly, an embedding tensor is mentioned but never introduced. It kind of becomes clearer in the following description, but this description is too short and not motivated. We don’t know the goal of this part of the paper, and we discover the objectives while reading it. The goal is to estimate outputs for each instance in the scene, which requires identifying the object instances first, done by clustering. This should be better motivated and related to the embedding tensor.  Metric learning is thrown into the text in a quick way, referencing DRLIM (reference 11). In order to make the paper self-contained, this should be properly developed.  Finally, all the different losses should be given correctly with equations. In particular, how does the method deal with the fact that multiple outputs are created for the scene (the different discovered object instances) and need to be matched to the multiple ground truth objects? In the 2D object detection literature, this is tackled in different ways, either using bi-partite graph matching / the Hungarian algorithm (Multi-box) or through a grid-based heuristic (YOLO, SSD).  The experiments are convincing and show good performance against strong baselines.  On a minor note, “random” and “oneway” are only shown in figure 3 and not figure 2.  === After rebuttal  === After rebuttal  I keep my positive rating, since the rebuttal addressed some of my concerns. However, the following 4 references are very close to the paper and should be cited and properly referenced:  - MapNet (CVPR 18): http://www.robots.ox.ac.uk/~joao/mapnet/ - IQA (CVPR 18): https://pjreddie.com/media/files/papers/IQA.pdf - Im2Pano3D (CVPR 18): https://arxiv.org/pdf/1712.04569.pdf - Neural Map (ICLR 2017): https://arxiv.org/abs/1702.08360


Bandit Learning with Positive Externalities
In many platforms, user arrivals exhibit a self-reinforcing behavior: future user arrivals are likely to have preferences similar to users who were satisfied in the past. In other words, arrivals exhibit positive externalities. We study multiarmed bandit (MAB) problems with positive externalities. We show that the self-reinforcing preferences may lead standard benchmark algorithms such as UCB to exhibit linear regret. We develop a new algorithm, Balanced Exploration (BE), which explores arms carefully to avoid suboptimal convergence of arrivals before sufficient evidence is gathered. We also introduce an adaptive variant of BE which successively eliminates suboptimal arms. We analyze their asymptotic regret, and establish optimality by showing that no algorithm can perform better.

In this paper, the authors study the effects of positive externalities in MAB. In real platforms such as news site, the positive externalities come from the fact that if one of the arms generates rewards, then the users who prefer that arm become more likely to arrive in the future. If the platform knows the user preferences or could infer it from data, the usual way for handling positive externalities is to use contextual bandits: the services are personalized for each kind of users. However, most of the time the contexts are not very informative about the user preferences.  The usual alternative is to consider that the reward process evolves during time, and hence to use adversarial bandits or piecewise stationary bandits. Here, the authors propose to take advantage of prior knowledge on how the reward process evolves during time: the positive externalities.   The positive externalities change the usual trade-off between exploration and exploitation. Indeed, the effects of the choices of the platform are amplified. If the platform chooses the optimal arm, then this choice is amplified by the arriving of the users that like the optimal arm. However, if the algorithm chooses a sub-optimal arm, the price to pay in terms of future rewards can be dramatic. In order to analyze the positive externality effect, the authors introduce the regret against an Oracle which knows the optimal arm.  Depending on the value of \alpha, which measures the strength of positive externalities, a regret lower bound of MAB with positive externalities is provided. Then they bring out that classical approaches are suboptimal. Firstly, they show that UCB algorithm achieves a linear regret for bandits with positive externalities. Secondly, they show that explore then exploit algorithm may incur a linear regret when \alpha > 1. A first algorithm called Balanced Exploration (BE) is introduced. In the exploration phase the arm which has the lowest cumulated reward is chosen, while in the exploitation phase the arm which has been the less chosen in the exploration phase is played. it is worth noting that BE algorithm needs only to know the time horizon. The analysis of this algorithm shows that it is near optimal. A second algorithm, that assumes that in addition to the time horizon the parameters of positive externalities are known (\alpha and \theta_a), is proposed. This algorithm uses an unbiased estimator of the mean reward thanks to the knowledge of the arrival probability. The algorithm plays the arm that has received the least rewards in order to explore. In order to exploit, the suboptimal arms are eliminated when their upper bounds are lesser that the lower bound of the estimated best arm. The authors show that Balanced Exploration with Arm Elimination is optimal.  I regret that there are no experiments because it could be interesting for the reader to observe the gap in performances between BE, where the arrival probabilities are unknown and BE-AE, where the arrival probabilities are known.  Overall, the paper is very interesting and I vote for acceptance.   Minor Comments: The notation \theta^\alpha line 151 is misleading. You use the notation power alpha for N_a^\alpha and for \theta^\alpha but it does not mean the same thing. I suggest to use \theta(\alpha). Line 196, there is a typo u_a(0) = \infty and not 0.


Continuous-time Value Function Approximation in Reproducing Kernel Hilbert Spaces
Motivated by the success of reinforcement learning (RL) for discrete-time tasks such as AlphaGo and Atari games, there has been a recent surge of interest in using RL for continuous-time control of physical systems (cf. many challenging tasks in OpenAI Gym and DeepMind Control Suite). Since discretization of time is susceptible to error, it is methodologically more desirable to handle the system dynamics directly in continuous time. However, very few techniques exist for continuous-time RL and they lack flexibility in value function approximation. In this paper, we propose a novel framework for model-based continuous-time value function approximation in reproducing kernel Hilbert spaces. The resulting framework is so flexible that it can accommodate any kind of kernel-based approach, such as Gaussian processes and kernel adaptive filters, and it allows us to handle uncertainties and nonstationarity without prior knowledge about the environment or what basis functions to employ. We demonstrate the validity of the presented framework through experiments.

Strengths 1. Considering dynamic programming problems in continuous time such that the methodologies and tools of dynamical systems and stochastic di_x000b_eren- tial equations is interesting, and the authors do a good job of motivating the generalities of the problem context.  2. The authors initially describe problems/methods in very general terms, which helps preserve understandability of the _x000c_first sections.  Weaknesses  1. The parameterizations considered of the value functions at the end of the day belong to discrete time, due to the need to discretize the SDEs and sample the state-action-reward triples. Given this discrete implementa- tion, and the fact that experimentally the authors run into the conven- tional di_x000e_culties of discrete time algorithms with continuous state-action function approximation, I am a little bewildered as to what the actual bene_x000c_t is of this problem formulation, especially since it requires a re- de_x000c_nition of the value function as one that is compatible with SDEs (eqn. (4) ). That is, the intrinsic theoretical bene_x000c_ts of this perspective are not clear, especially since the main theorem is expressed in terms of RKHS only.  2. In the experiments, the authors mention kernel adaptive _x000c_filters (aka kernel LMS) or Gaussian processes as potential avenues of pursuit for addressing the function estimation in continuous domains. However, these methods are fundamentally limited by their sample complexity bottleneck, i.e., the quadratic complexity in the sample size. There's some experimental ref- erence to forgetting factors, but this issue can be addressed in a rigorous manner that preserves convergence while breaking the bottleneck, see, e.g.,  A. Koppel, G. Warnell, E. Stump, and A. Ribeiro, ``Parsimonious on- line learning with kernels via sparse projections in function space," arXiv preprint arXiv:1612.04111, 2016.  Simply applying these methods without consideration for the fact that the sample size conceptually is approaching in_x000c_nity, makes an update of the form (16) inapplicable to RL in general. Evaluating the Bellman operator requires computing an expected value.  3. Moreover, the limited complexity of the numerical evaluation is reflective of this complexity bottleneck, in my opinion. There are far more effective RKHS value function estimation methods than GPTD in terms of value function estimation quality and memory efficiency:  A. Koppel, G. Warnell, E. Stump, P. Stone, and A. Ribeiro. ``Policy Evaluation in Continuous MDPs with E_x000e_cient Kernelized Gradient Tem- poral Di_x000b_fference," in IEEE Trans. Automatic Control (submitted), Dec. 2017."  It's strange that the authors only compare against a mediocre benchmark rather than the state of the art.  4. The discussion at the beginning of section 3 doesn't make sense or is written in a somewhat self-contradictory manner. The authors should take greater care to explain the di_x000b_erence between value function estimation challenges due to unobservability, and value function estimation problems that come up directly from trying to solve Bellman's evaluation equation. I'm not sure what is meant in this discussion.  5. Also, regarding L87-88: value function estimation is NOT akin supervised learning unless one does Monte Carlo rollouts to do empirical approxima- tions of one of the expectations, due to the double sampling problem, as discussed in  R. S. Sutton, H. R. Maei, and C. Szepesvari, \A convergent o(n) temporal- di_x000b_erence algorithm for o_x000b_-policy learning with linear function approxi- mation," in Advances in neural information processing systems, 2009, pp. 1609?1616.   and analyzed in great detail in :  V. R. Konda and J. N. Tsitsiklis, ``Convergence rate of linear two-timescale stochastic approximation," Annals of applied probability, pp. 796- 819, 2004.  6. The Algorithm 1 pseudo-code is strangely broad so as to be hand-waving. There's no speci_x000c_cs of a method that could actually be implemented, or even computed in the abstract. Algorithm 1 could just as well say "train a deep network" in the inner loop of an algorithm, which is unacceptable, and not how pseudo-code works. Specifically, one can't simply "choose" at random" an RKHS function estimation algorithm and plug it in and assume it works, since the lion-share of methods for doing so either re- quire in_x000c_nite memory in the limit or employ memory-reduction that cause divergence.  7. L107-114 seems speculative or overly opinionated. This should be stated as a remark, or an aside in a Discussion section, or removed.  8. A general comment: there are no transitions between sections, which is not good for readability.  9. Again, the experiments are overly limited so as to not be convincing. GPTD is a very simplistic algorithm which is not even guaranteed to pre- serve posterior consistency, aka it is a divergent Bayesian method. There- fore, it seems like a straw man comparison. And this comparison is con- ducted on a synthetic example, whereas most RL works at least consider a rudimentary OpenAI problem such as Mountain car, if not a real robotics, power systems, or _x000c_financial application.


Gradient Descent Meets Shift-and-Invert Preconditioning for Eigenvector Computation
Shift-and-invert preconditioning, as a classic acceleration technique for the leading eigenvector computation, has received much attention again recently, owing to fast least-squares solvers for efficiently approximating matrix inversions in power iterations. In this work, we adopt an inexact Riemannian gradient descent perspective to investigate this technique on the effect of the step-size scheme. The shift-and-inverted power method is included as a special case with adaptive step-sizes. Particularly, two other step-size settings, i.e., constant step-sizes and Barzilai-Borwein (BB) step-sizes, are examined theoretically and/or empirically. We present a novel convergence analysis for the constant step-size setting that achieves a rate atÕ(
, where λ i represents the i-th largest eigenvalue of the given real symmetric matrix and p is the multiplicity of λ 1 . Our experimental studies show that the proposed algorithm can be significantly faster than the shift-and-inverted power method in practice.

This paper proposes an algorithm which combines shift-and-invert preconditioning and gradient descent to compute leading eigenvector. Shift-and-invert preconditioning is a common preconditioning technique, and gradient descent is one of the most popular optimization algorithms, so this combination seems reasonable. They establish the complexity of using this method, by using a somewhat new potential function.  Pros: The main difference of this paper with previous works is that it allows the multiplicity of the leading eigenvalue to be larger than 1, while most works assume an eigen-gap between the largest and second largest eigenvalue.          In addition, this paper introduces a somewhat new potential function, that depends on the angle between a vector and the subspace spanned by the eigenvectors. More specifically, the proof uses two potential functions, one is log(u) and another is 1 – sin^2(v), where u and v depend on the angle. The second one is not new, but the first one log(.) seems to be new.    Cons: There are quite a few issues to be addressed/clarified.  1.  Complexity.   Perhaps the biggest issue is: the complexity proved in this work is not shown to be better than the other existing results (a bit subtle; see explanation below). In fact, we need to discuss two cases:   Case 1: the multiplicity of the largest eigenvalue is 1, i.e $\lambda_1 > \lambda_2 = \lambda_{p+1}$. In this case, the complexity of this work is not better than those in [Musco and Musco, 2015] and [Garber et al., 2016] (in fact, the same).   Case 2: the multiplicity of the leading eigenvalue is larger than 1. In this case, the proposed rate is the “best“ so far. However, most other works do not consider this scenario, so this is not a fair comparison.   Overall, the improvement only in Case 2 seems to be a minor progress.   BTW: in Table 1, it seems that $\sqrt{\lambda_1}$ is missing from the numerator of the rate for ``this work''.   2. Limited Novelty of Proposed Algorithm    There are already a few works which combine shift-and-invert preconditioning with some common base methods, such as accelerated gradient descent [Wang et al., 2017], accelerated randomized coordinate descent methods [Wang et al., 2017] or SVRG [Garber et al., 2016]. It is not surprising to use gradient descent (GD) for subproblems. From the arguments made in paper, I fail to see why GD will be a better choice.          First, the authors claim that GD is better because Lanczos algorithms does not work for positive definite matrix, but GD does (line 39). However, some other base methods, such as``coordinate-wise'' methods [Wang et al., 2017], also work for positive definite matrices.         Second, the authors claim GD will not be stuck at saddle points with probability one from a random starting point. The authors make this point by citing [Pitaval et al., 2015] (line 42). However, [Pitaval et al., 2015] is about a quite different problem. In addition, people tend to believe all reasonable first order methods will not be stuck at saddle points.          The warm-start technique (line 158) is also not new to the community. The same formulation of warm-start has been used in [Garber et al., 2016].  3 Algorithm Details         The first negative sign in line 4 of Algorithm 1 should be removed. From equation (3), Riemannian gradient is equal to Euclidean gradient left multiplied by $(I - x_{t-1} x_{t-1}^T)$. In Algorithm 1, $y_t$ is the approximate Euclidean gradient. Therefore, the approximation Riemannian gradient in line 4 should be $(I - x_{t-1} x_{t-1}^T) y_t$.         Another thing that requires clarification is the choice of stepsize. In Algorithm 1, authors pick the stepsize at every iteration such that the stepsize satisfies the second inequality in line 176. Why do the authors pick the stepsize at each iteration $t$, while the second inequality in line 176 is not dependent on each iteration $t$. Why don't we pick a constant stepsize before running the iterations?   4. Experiments The authors only compare Rimennian gradient descent solver with/without shift-and-invert preconditioning, and shift-and-inverted power method. As I mentioned before, the authors does not show the complexity of this work is better than [Musco and Musco, 2015] (Block Krylov) and [Garber et al., 2016] (shift-and-inverted power method). Therefore, it is reasonable to also consider comparing with these methods.  5. Minor issues:   --It seems that the authors did not explicitly define Riemannian gradient.    -- line 171 should be Lemma 3.2 rather than 3.3.  After the rebuttal: I've increased my score by one, due to the good empirical performance. The tone of this paper is like "our contribution is to combine two methods, and get the best rate so far", and my above comments said, "combining two methods is rather straightforward, and the rate does not improve previous ones" --thus a clear reject. If the authors motivate the paper from an empirical perspective by saying "we find the combination of SI+GD works really well, so to promote it we provide theoretical justification", the contribution may be more clear. Considering the above reasons, I think this paper is slightly below the acceptance threshold.  [Garber et al., 2016] Dan Garber, Elad Hazan, Chi Jin, Sham M. Kakade, Cameron Musco, Praneeth Netrapalli, and Aaron Sidford. Faster eigenvector computation via shift-and-invert preconditioning. In International Conference on Machine Learning, pages 2626–2634, 2016.  [Musco and Musco, 2015] Cameron Musco and Christopher Musco. Randomized block krylov methods for stronger and faster approximate singular value decomposition. In Advances in Neural Information Processing Systems, pages 1396–1404, 2015.  [Pitaval et al., 2015] Renaud-Alexandre Pitaval, Wei Dai, and Olav Tirkkonen. Convergence of gradient descent for low-rank matrix approximation. IEEE Trans. Information Theory, 61(8):4451–4457, 2015.  [Wang et al., 2017] Jialei Wang, Weiran Wang, Dan Garber, and Nathan Srebro. Efficient coordinate-wise leading eigenvector computation. CoRR, abs/1702.07834, 2017. URL http://arxiv.org/abs/1702. 07834.


Diversity-Driven Exploration Strategy for Deep Reinforcement Learning
Efficient exploration remains a challenging research problem in reinforcement learning, especially when an environment contains large state spaces, deceptive or sparse rewards. To tackle this problem, we present a diversity-driven approach for exploration, which can be easily combined with both off-and on-policy reinforcement learning algorithms. We show that by simply adding a distance measure regularization to the loss function, the proposed methodology significantly enhances an agent's exploratory behavior, and thus prevents the policy from being trapped in local optima. We further propose an adaptive scaling strategy to enhance the performance. We demonstrate the effectiveness of our method in huge 2D gridworlds and a variety of benchmark environments, including Atari 2600 and MuJoCo. Experimental results validate that our method outperforms baseline approaches in most tasks in terms of mean scores and exploration efficiency.

[Summary]: The paper proposes a formulation for incentivizing efficient exploration in deep reinforcement learning by encouraging the diversity of policies being learned. The exploration bonus is defined as the distance (e.g., KL-divergence) between the current policy and a set of previous policies. This bonus is then added to the standard reinforcement learning loss, whether off-policy or on-policy. The paper further uses an adaptive scheme for scaling the exploration bonus with respect to external reward. The first scheme, as inspired from [Plappert et.al. 2018], depends on the magnitude of exploration bonus. The second scheme, on the other hand, depends on the performance of policy with respect to getting the external reward.  [Paper Strengths]: The paper is clearly written with a good amount of details and is easy to follow. The proposed approach is intuitive and relates closely to the existing literature.  [Paper Weaknesses and Clarifications]: - My major concerns are with the experimental setup:     (a) The paper bears a similarity in various implementation details to Pappert et.al. [5] (e.g. adaptive scaling etc.), but it chose to compare with the noisy network paper [8]. I understand [5] and [8] are very similar, but the comparison to [5] is preferred, especially because of details like adaptive scaling etc.     (b) The labels in Figure-5 mention that DDPG w/ parameter noise: is this method from Plappert et.al. [5] or Fortunato et.al. [8]. It is unclear.     (c) No citations are present for the names of baseline methods in the Section-4.3 and 4.4. It makes it very hard to understand which method is being compared to, and the reader has to really dig it out.     (d) Again in Figure-5, what is "DDPG(OU noise)"? I am guessing its vanilla DDPG. Hence, I am surprised as to why is "DDPG (w/ parameter space noise)" is performing so much worse than vanilla DDPG? This makes me feel that there might be a potential issue with the baseline implementation. It would be great if the authors could share their perspective on this.     (e) I myself compared the plots from Figure-1,2,3, in Pappert et.al. [5] to the plots in Figure-5 in this paper. It seems that DDPG (w/ parameter space noise) is performing quite worse than their TRPO+noise implementation. Their TRPO+noise beats the vanilla TRPO, but DDPG+noise seems to be worse than DDPG itself. Please clarify the setup.     (f) Most of the experiments in Figure-4 seems to be not working at all with A2C. It would be great if authors could share their insight. - On the conceptual note: In the paper, the proposed approach of encouraging diversity of policy has been linked to "novelty search" literature from genetic programming. However, I think that taking bonus as KL-divergence of current policy and past policy is much closer to perturbing policy with a parameter space noise. Both the methods encourage the change in policy function itself, rather than changing the output of policy. I think this point is crucial to the understanding of the proposed bonus formulation and should be properly discussed. - Typo in Line-95  [Final Recommendation]: I request the authors to address the clarifications and comments raised above. My current rating is marginally below the acceptance threshold, but my final rating will depend heavily on the rebuttal and the clarifications for the above questions.  [Post Rebuttal] Reviewers have provided a good rebuttal. However, the paper still needs a lot of work in the final version. I have updated my rating.


Stimulus domain transfer in recurrent models for large scale cortical population prediction on video
To better understand the representations in visual cortex, we need to generate better predictions of neural activity in awake animals presented with their ecological input: natural video. Despite recent advances in models for static images, models for predicting responses to natural video are scarce and standard linear-nonlinear models perform poorly. We developed a new deep recurrent network architecture that predicts inferred spiking activity of thousands of mouse V1 neurons simultaneously recorded with two-photon microscopy, while accounting for confounding factors such as the animal's gaze position and brain state changes related to running state and pupil dilation. Powerful system identification models provide an opportunity to gain insight into cortical functions through in silico experiments that can subsequently be tested in the brain. However, in many cases this approach requires that the model is able to generalize to stimulus statistics that it was not trained on, such as band-limited noise and other parameterized stimuli. We investigated these domain transfer properties in our model and find that our model trained on natural images is able to correctly predict the orientation tuning of neurons in responses to artificial noise stimuli. Finally, we show that we can fully generalize from movies to noise and maintain high predictive performance on both stimulus domains by fine-tuning only the final layer's weights on a network otherwise trained on natural movies. The converse, however, is not true.

##### I have read over the author's rebuttal, and I am satisfied by their clarification and response. ##### In this submission, the authors use modern deep learning techniques to develop a predictive model of mouse V1 cortex. Most notably, this model seeks to fill a gap in the literature for simultaneously predicting large populations of neurons responding to both natural video and noise stimuli, whereas previous models had mainly focused on static image stimuli in small cortical populations.   The model developed by the authors receives these video stimuli as inputs, and consists of well-established computer vision architectures — here a feed-forward convolutional network feeding into a convolutional GRU — to extract stimulus feature representations common to all neurons. These are then read-off and modulated by a spatial-transformer network to ensure receptive-field stability of the regressed neural outputs. A novel structural contribution has been added to account for non-stimulus driven experimental noise which might be present in the data. Specifically, the readouts are additionally modified by “shift” and “modulator” networks which factor in these sources of noise such as pupil movement (correction to this is needed for proper receptive-field mapping), pupil diameter, and measured running speed of the animal (which is known to be correlated with V1 activity in rodents).  They show that this model outperforms a non-recurrent linear-nonlinear control model (fig. 2a) across both datasets for all three mice, and assess the relative contributions of the novel shift and modulator component networks (fig. 2c). In “domain transfer” experiments, they show both qualitative and quantitative evidence for models generalizing outside their training set to produce receptive fields and tuning curves similar to mouse V1 activity, even though the mice have been shown different noise stimuli. Lastly, the authors study the quantitative amount of transfer between stimulus domains and find that models trained on movies serve as reasonable predictors of white-noise induced activity, and find that the converse is not true (fig. 4b). However, a network which combined domain knowledge from both stimuli sets provides a sufficient basis space for explaining activity presented in either domain (fig. 4c).  Overall, I am impressed by the ambition of this work. I believe that the field could benefit from more of this style of large-scale modeling, and further consideration of generalization to novel stimuli as a complex yet intriguing phenomenon. The application to large-scale populations responding to complex stimuli, instead of only a small subset of V1 responding to simpler stimuli is also noteworthy. I found the introduction of the shifter and modulator networks to be a novel idea, encoding well-thought-out structural priors into the model.  That being said, although I have no complaints concerning the high-level scope of analysis presented in this submission, I do have some concerns as to the strength of results for several reasons:  (1). I am willing to believe the qualitative comparison between the “full” model to a “standard” linear-nonlinear system in fig 2a. Based on the lack of recurrence alone, the result seems intuitive. However, I am concerned with the quantitative estimate between these two models due to not controlling for numbers of learnable parameters. From what I understand, the authors balance the number of parameters in the ConvNet betweeen models — three 12-feature layers in the full model and one 36-feature layer in the LN mode — and readout structures. However, in the absence of the GRU in the LN model, there is an imbalance in these numbers. Generally speaking, you can interpret this in two ways: either (i). the extra units and nonlinearities can artificially inflate the score of the full model if no hyperparameter optimization has been performed or (ii) the authors had performed a hyper parameter sweep (number of units) across both models and settled on the best performing settings. Given that no mention towards the latter has been discussed in the submission, I am inclined to believe that the former is true.  (2). I am concerned with a statement that was made inside the “Network Implementation” subsection which stated that all readouts biases were initialized to the mean firing rate of that neuron. It seems to me that this choice is also reasonable, however it sounds like that is a strong prior for what these values should be. Could the authors discuss the impact of this initialization, and how it relates to why the full model might only be performing ~50-70% of the oracle estimator?  (3). Similar to the above in (1), even if I am intrigued by the application of the shift and modulator networks, I am concerned that for parameter balancing reasons and the lack of a hyperparameter sweep, the results in 2c might be artificially inflated. Additionally, I’m concerned that the novelty of such an approach might be detracting from what might be a more standard way of incorporating these stimulus-independent noise sources — namely, by having them as generic features that are concatenated as inputs to the readout MLP, rather than these specific priors. I would be just as impressed by seeing a larger/deeper readout network with a more generic structure perform at the level (or better) than something with a “clever” implementation.   (4). Almost unsurprisingly, the authors find that models trained on movies are well-correlated with noise-induced activity, although the converse is not true, and that a model trained on both can in fact predict both. I would have liked a similar comparison between natural movies and the rendered videos as well, because it is unclear whether the statistics between even these two domains are imminently transferable. What I did find interesting here is that although the movie-optimized models are well correlated with noise responses, they show high variance in predicting the low-levels details such as orientation and direction tuning. The paper ends on a bit of a cliff-hanger in the sense that they do not propose a better method of a generalized model than what amounts to just training on all stimulus contingencies.   In summary, I am inclined to like this work. The results — although not necessarily as strong as they could be — feel intuitive. If text length were not a factor, I personally feel like I would have benefitted from an extended discussion/interpretation section of results, and more detailed analysis of those presented in the submission (specifically as far as the domain transfer section is concerned). The approach and aims of this work alone are a worthwhile contribution to the neuroscience community as a whole.


BML: A High-performance, Low-cost Gradient Synchronization Algorithm for DML Training
In distributed machine learning (DML), the network performance between machines significantly impacts the speed of iterative training. In this paper we propose BML, a new gradient synchronization algorithm with higher network performance and lower network cost than the current practice. BML runs on BCube network, instead of using the traditional Fat-Tree topology. BML algorithm is designed in such a way that, compared to the parameter server (PS) algorithm on a Fat-Tree network connecting the same number of server machines, BML achieves theoretically
dominating communication cost. Therefore, in order to run DML in large scale, we need to carefully design the network with minimized synchronization overhead among machines.
The widely-used network topology to run DML in today's data centers is Clos network, or FatTree [5]. Although Fat-Tree achieves great success in providing uniform network performance to cloud computing applications, it may not well match the traffic model of gradient synchronization in DML. Running the typical parameter server (PS) synchronization algorithm in Fat-Tree, each synchronization flow needs to traverse multiple hops of switches before being aggregated. It not only hurts the gradient synchronization performance, but also wastes the bandwidth/link resource.
In this paper, we suggest using BCube [12] as the underlying network topology for DML training, and design a novel distributed gradient synchronization algorithm on top of BCube, called BML. BCube is a recursive network topology composed of commodity switches and servers with k (the typical value of k is 2∼4) interfaces. The synchronization algorithm of BML is designed in such a way that, compared to the PS algorithm running a Fat-Tree network connecting the same number of server machines, BML running on a BCube network can theoretically achieve 1 k of the gradient synchronization time, with only k 5 of switches. We have implemented BML in TensorFlow. We run two representative public deep learning benchmarks, namely, LeNet-5 [15] and VGG-19 [22], on a testbed with 9 dual-GPU servers. The experiment results show that, BML can reduce the job completion time of DML training by up to 56.4% compared with the PS algorithm on Fat-Tree network. The advantage of BML is higher when the sub-minibatch size per machine is smaller, which is important for large-scale DML to guarantee the model accuracy.
2 Background and Motivation DML Models and Notations: DML can run on multiple CPUs/GPUs in a machine or on multiple machines. In this work we focus on DML network among machines. In order to decouple the intermachine and intra-machine communications, throughout this paper we simply take one machine as a single training worker, though the machine can be equipped with multiple GPUs.
Based on splitting whether the training data or the model parameters onto multiple machines, DML can be divided into data-parallel and model-parallel ones. In data-parallel DML, each machine uses a shard of training data to compute the gradients; while in model-parallel DML, a machine computes gradients for part of the model. In this work we focus on data-parallel DML. In each iteration, every machine trains local gradients for the entire model based on its sub-minibatch of training data, and synchronizes the gradients with other machines. The aggregated gradients are calculated upon all the machines' local gradients, which are then applied to the model update.
According to the tradeoff between gradient freshness and computing resource utilization, there are three typical synchronization modes. 1) Bulk synchronous parallel (BSP). 2) Total asynchronous parallel (TAP). 3) Stale synchronous parallel (SSP). Given a predefined accuracy of the trained model, it is difficult to tell which synchronization mode runs the fastest in practice. BSP wastes the computation resource of some faster machines, but fully follows the sequential behavior as trained by a single machine. TAP makes full utilization of the computing resource, but the convergence speed is unpredictable with the possibility of no convergence at all [10]. SSP lies between the two with proven convergence [9,26]. In this work we focus on BSP synchronization, which is widely used in modern ML applications [11,24].

This paper observes that by putting network cards into computers, some of the router functionality can be performed for free by each compute node.  This potentially means less router equipment, and less transmission time for averaging data.  This paper then shows how to you how to do it for real, giving details of the averaging algorithm for 'k' network cards per compute node. For k=2 and small minibatches, they actually see their expected factor of 2 speedup experimentally. The algorithm has some little tricks in it that make it a useful contribution.   It is nice that this technique can in many cases be combined with other ways to reduce communication overhead. The technique is more something to consider when setting up a new compute cluster.  This is really a networking paper, since the technique describes how to aggregate [sum] and distribute any vector amongst N compute nodes. The vector could be a gradient, a set of model parameters, or, well, anything else. On the other hand, they actually demonstrate experimentally that it can work in practice for a fairly common machine learning scenario.  ### After author review ### Margin notes: line 95:    GST -  where defined? line 226: codes --> code  Some of the statements in the rebuttal seemed more understandably stated than the more difficult level of presentation in the paper. Occasionally I had to remember  previous statements and infer comparisons by myself.  As noted by reviewer #2, the comparison figure and its discussion helps bring out the practical importance of their paper.  I agree with reviewer #1 that Figure 4 and its discussion needs a very devoted reader.  If I had to make room, I might consider moving Figure 4, lines 165--203 and the dense notation use for it (148--157) to a Supplementary Material.  The body of the paper might contain a much simplified version of Fig. 4 and its discussion, with just the thread 0,1 link activity during the two steps each of aggregation and broadcast, and refer the reader to the supplementary material for details of precisely which gradient piece is chosen for transmission so as to lead to a final state where all servers know all pieces.


Transfer of Deep Reactive Policies for MDP Planning
Domain-independent probabilistic planners input an MDP description in a factored representation language such as PPDDL or RDDL, and exploit the specifics of the representation for faster planning. Traditional algorithms operate on each problem instance independently, and good methods for transferring experience from policies of other instances of a domain to a new instance do not exist. Recently, researchers have begun exploring the use of deep reactive policies, trained via deep reinforcement learning (RL), for MDP planning domains. One advantage of deep reactive policies is that they are more amenable to transfer learning. In this paper, we present the first domain-independent transfer algorithm for MDP planning domains expressed in an RDDL representation. Our architecture exploits the symbolic state configuration and transition function of the domain (available via RDDL) to learn a shared embedding space for states and state-action pairs for all problem instances of a domain. We then learn an RL agent in the embedding space, making a near zero-shot transfer possible, i.e., without much training on the new instance, and without using the domain simulator at all. Experiments on three different benchmark domains underscore the value of our transfer algorithm. Compared against planning from scratch, and a state-of-the-art RL transfer algorithm, our transfer solution has significantly superior learning curves.

(i) Summary:  The paper proposes TransPlan, a novel neural architecture dedicated to “near” zero-shot transfer learning between equi-sized discrete MDP problems from the same RDDL domain (i.e., problems with same number of state and action variables, but with different non-fluents / topologies). It combines several deep neural models to learn state and state-action embeddings which shall be used to achieve sample-efficient transfer learning across different instances. It uses GCNs (Graph Convolutional Networks), a generalization of ConvNets for arbitrary graph embeddings, to learn state embeddings, A3C (Asynchronous Advantage Actor Critic) as its RL module that learns policies mapping state embeddings to an abstract representation of probability distributions over actions, and action decoders that attempt to transform these abstract state-action embeddings into the MDP’s original action space for the target problem. Experiments on IPPC domains (e.g., SysAdmin, Game of Life, and Navigation) compare the proposed approach with A3C (without transfer learning) and A2T (Attend, Adapt and Transfer). Results indicate that TransPlan can be effective for transfer learning in these domains.  (ii) Comments on technical contributions:  - Contributions seem a bit overstated: The authors argue that the proposed transfer learning approach is the “first domain-independent transfer algorithm for MDP planning domains expressed in RDDL”. Regardless of whether TransPlan is the first transfer algorithm for the RDDL language, the proposed approach does not seem “domain-independent” in a very general sense, given that its state encoder relies on RDDL domains exhibiting a graph structure (i.e., “state variables as nodes, edges between nodes if the respective objects are connected via the non-fluents in the domain”) - which explains the particular choice of IPPC domains for experiments. Moreover, the transfer learning is only focused on target instances with same size as the instances used in the learning phase, which seems like an important limitation. Finally, it is not clear until the experiments section that TransPlan is only applicable to discrete domains, another limitation to a subset of RDDL.  - The paper contributions are not put in context with recent related work: The main works related to the idea of Deep Reactive Policies in model-known planning (e.g., “Action schema networks: Generalised policies with deep learning”, and “Training deep reactive policies for probabilistic planning problems”) are barely described and not discussed at all. In particular, it is not even mentioned that Action-Schema Nets also address the transfer learning between instances of the same model independently of the size of the target instances.  Is the RDDL vs. PPDDL issue that important that these methods are incomparable?  - The background necessary to appreciate the proposed method could be presented better.  Section 2.1. Reinforcement Learning  The presentation of the Actor-Critic architecture (which is the core aspect of A3C) is really confusing. First off, it is said that A3C “constructs approximations for both the policy (using the ‘critic’ network) and the value function (using the ‘actor’ network)”. This is wrong or at least very unclear. The actor network is the approximation for the policy and the critic network it is used as the baseline in the advantage function (i.e., the opposite of what the authors seem to say). Moreover, the formulation of the policy gradient seems totally wrong as the policy \pi(a|s) is the only  \theta-parameterized function in the gradient (it should be denoted instead as \pi_{\theta}(a|s)), and also the value functions Q_{\pi}(s, a; \theta) and V(s; \theta) do not share the same set of parameters - indeed, Q-values are approximated via Monte-carlo sampling and V(s; \theta) is the baseline network learned via regression. Additionally, the actor network does not maximize the H-step lookahead reward by minimizing the expectation of mean squared loss, the minimization of the mean squared loss is part of the regression problem solved to improve the approximation of the critic network (i.e., the baseline function); it is the policy gradient that attempts to maximize the expected future return.  Section 2.2. Probabilistic Planning  The presentation of the RDDL language is not very clear and could mislead the reader.  Aside from technical imprecision, the way it is presented can give the reader the impression that the language is supposed to be solely oriented to “lifted” MDPs, whereas I understand that it is a relational language aimed at providing a convenient way to compactly represent ground factored MDPs problems.  This intent/usage of RDDL should be clarified.  Section 2.3 Graph Convolutional Networks  - The formulas are not consistent with the inputs presented: The explanation of the inputs and outputs of the GCN layer in lines 149-151 are confusing since parameter N is not explained at all and D^{(l+1)} is simply not used in the formula for F^{(l+1)}.  - The intuition for the GCN layer and propagation rule/formula is too generic: The formula for the GCN layer’s activation function has not meaning at all for anyone not deeply familiar with ICLR2017 paper on Semi-supervised classification with graph convolutional networks. Also, the intuition that “this propagation rule implies that the feature at a particular node of the (l + 1)th layer is the weighted sum of the features of the node and all its neighbours at the lth layer” is completely generic and not does not elucidate at all the formula in line 152. Finally, the statement that “at each layer, the GCN expands its receptive field at each node by 1” is not explained and difficult to understand for someone not expert in GCNs.  - Problem formulation is not properly formalized: The authors say that “we make the assumption that the state and action spaces of all problems is the same, even though their initial state, non-fluents and rewards could be different”. Putting together this statement with the following statement in lines 95-96 “Our work attempts a near-zero shot learning by learning a good policy with limited learning, and without any RL.” triggers some questioning not explicitly addressed in the paper about how could the rewards be different in the target instances and the transfer learning still be effective without any RL. The authors should really clarify what they mean by the “rewards could be different”.  (iii) Comments on empirical evaluation:  The empirical evaluation is centered around answering the following questions: (1) Does TRANSPLAN help in transfer to new problem instances? (2) What is the comparison between TRANSPLAN and other state of the art transfer learning frameworks? (3) What is the importance of each component in TRANSPLAN?  Questions (1) and (2) are partially answered by the experiments. Indeed, TransPlan does seem to transfer the learning to new problem instances (given the limitations pointed earlier) and it improves on the transfer learning A2T approach. But these conclusions are based solely on “learning curves” plotted in terms of “number of iterations”. So, it is important to remember that the first and foremost motivation of transfer learning is to amortize the computational cost of training the neural model over all target instances of the domain. If this is not successfully accomplished, there is not point in incurring the cost of the offline learning phase, and then it is best to plan from scratch for each instance of the domain. Therefore, it is in my understanding that a better experimental design should focus on measuring and comparing learning/training times and transfer times, instead of relying on the number of iterations to showcase the learning evolution of TransPlan. Particularly, this is important to fairly highlight the value of transfer learning when comparing with baseline A3C, and the particular advantages of TransPlan when comparing with the model-free A2T transfer learning approach.  Additionally, the authors should better explain why in Table 1 the columns for iter 0 are not 0.0 and the last columns for iter \infty are not 1.0. Judging by the formula of \alpha(i) in line 300, this should be the case.  (iv) Detailed comments:  Typos: - Line 27: “[...] and RDDL planning solve each problem instance [...]” should be “[...] and RDDL planning that solve each problem instance [...]” - Line 85: it lacks the proper reference for “transferring the value function or policies from the source to the target task []” - Line 109: “RDDL Reprentation” should be “RDDL Representation”  Other suggestions for better clarity: - Lines 107-108: “[...] an MDP exponential states [...]” should be “[...] an MDP exponential state space [...]” - Line 123: “[...] this domain is inherently non-deterministic [...]” would be better phrased “[...] this domain is inherently dynamic [...]”? - Figure 1: The model architecture is not really clear. Perhaps it would be better if the neural components used in the training/learning and transfer phases were separated with its inputs and outputs clearly shown in the figure. Also, an algorithm detailing the training procedure steps would greatly improve the presentation of the TransPlan approach.  (v) Overall evaluation:  The paper is relevant for the NIPS audience. It brings together interesting and novel ideas in deep reactive networks and training methods (i.e., domain-adversarial training). However, the overall readability of the paper is compromised by the unclear background section, the ambiguous presentation of the model architecture, and technical issues with some formulae. Additionally, the experimental setup seems biased to showcase what is stated by the authors in lines 314 - 315 “TRANSPLAN is vastly ahead of all algorithms at all times, underscoring the immense value our architecture offers to the problem.” In my understanding, it is a major experimental drawback to not compare the proposed approach with the baseline approaches w.r.t. the training/transfer times (instead, all comparisons are based on number of iterations), which is common in RL and transfer learning testbeds. So, my recommendation is to reject the paper mainly on the premises that the overall presentation must be improved for clarity/correctness; ideally the experiments should also take into consideration the computational costs in terms of training and transfer times.


Understanding Weight Normalized Deep Neural Networks with Rectified Linear Units
This paper presents a general framework for norm-based capacity control for L p,q weight normalized deep neural networks. We establish the upper bound on the Rademacher complexities of this family. With an L p,q normalization where q ≤ p * and 1/p+1/p * = 1, we discuss properties of a width-independent capacity control, which only depends on the depth by a square root term. We further analyze the approximation properties of L p,q weight normalized deep neural networks. In particular, for an L 1,∞ weight normalized network, the approximation error can be controlled by the L 1 norm of the output layer, and the corresponding generalization error only depends on the architecture by the square root of the depth.

This paper essentially seems to address 2 questions (a) Under certain natural weight constraints what is the Rademacher complexity upperbound for nets when we allow for bias vectors in each layer? and (b) How well can such weight constrained nets approximate Lipschitz functions in the sup norm over compact domains?   There is a section 4.1 in the paper which is about generalization bounds for nets doing regression. but to me that does not seem to be an essential part of the paper.   Let me first say at the outset that the writing of the paper seems extremely bad and many of the crucial steps in the proofs look unfollowable. As it stands this paper is hardly fit to be made public and needs a thorough rewriting!   There are few issues that I have with the section on upperbounding Rademacher complexity as has been done between pages 15 to 19,   1. Isnt the entire point of this analysis is just to show that the Srebro-Tomioka-Neyshabur result (http://proceedings.mlr.press/v40/Neyshabur15.pdf) holds even with biases? If that is the entire point then why is this interesting? I am not sure why we should have expected anything much to change about their de-facto exponential dependances on depth with addition of bias!     A far more interesting analysis that could have been done and which would have led to a far more interesting paper is if one could have lifted the Rakhlin-Golowich-Shamir analysis of Rademacher complexity (which does not have the exp(depth) dependance!) to nets with biases. I strongly feel thats the research that should have been done!   2. The writing of the proofs here are extremely difficult to follow and the only reason one can follow anything here is because one can often look up the corresponding step in Srebro-Tomioka-Neyshabur! There are just way too many steps here which make no sense to me : like what does "sup_{\x_i}" even mean in the lines below line 350? How did that appear and why is Massart's Lemma applicable for a sup over a continuum? It hardly makes any sense! I have no clue how equation 11(d) follows from equation 11(c). This is similar to Lemma 17 of Srebro-Tomioka-Neyshabur but this definitely needs a rederivation in this context. The steps following equation 12(b) also make little sense to me because they seem to indicate that one is taking \sigma of an innerproduct whereas this \sigma is I believe defined over vectors. Its again not clear as to how the composition structure below equation 12(d) got broken.    Now coming to the proof of Theorem 3 - its again fraught with too many problems. Lemma 9 is a very interesting proposition and I wish this proof were clear and convincing. As of now its riddled with too many ambiguous steps!   For example : below line 464 what is being computed are \floor(r_1/2) and \floor(r_2/2) number of linear functions apart from x and -x. Any linear function needs 2 ReLU gates for computation. But this doubling is not visible anywhere! Page 23 to 24, the argument gets even more difficult to follow :  I do not understand the point that is being made at the top of page 23 that there is somehow a difference between stating what the 2 hidden neurons are and what they compute. The induction is only more hairy and there is hardly any clear proof anywhere about why this composed function should be seen as computing the required function!   Maybe the argument of Theorem 3 is correct and it does seem plausible but as it stands I am unable to decide the correctness given how unintelligible the presentation is!    ============================================================== Response to "author response" :   I thank the authors for their detailed response. I do think its a very interesting piece of work and after the many discussions with the co-reviewers now I am more convinced of the correctness of the proofs. Though still there are major sections in the paper like the argument about Lemma 9/Theorem 3 which are still too cryptic for me to judge correctness with full certainty. This paper definitely needs a thorough rewriting with better notation and presentation.


Efficient Stochastic Gradient Hard Thresholding
Stochastic gradient hard thresholding methods have recently been shown to work favorably in solving large-scale empirical risk minimization problems under sparsity or rank constraint. Despite the improved iteration complexity over full gradient methods, the gradient evaluation and hard thresholding complexity of the existing stochastic algorithms usually scales linearly with data size, which could still be expensive when data is huge and the hard thresholding step could be as expensive as singular value decomposition in rank-constrained problems. To address these deficiencies, we propose an efficient hybrid stochastic gradient hard thresholding (HSG-HT) method that can be provably shown to have sample-size-independent gradient evaluation and hard thresholding complexity bounds. Specifically, we prove that the stochastic gradient evaluation complexity of HSG-HT scales linearly with inverse of sub-optimality and its hard thresholding complexity scales logarithmically. By applying the heavy ball acceleration technique, we further propose an accelerated variant of HSG-HT which can be shown to have improved factor dependence on restricted condition number in the quadratic case. Numerical results confirm our theoretical affirmation and demonstrate the computational efficiency of the proposed methods.

The article analyses convergence in hard-thresholding algorithms and proposes an accelerated stochastic hybrid hard thresholding method that displays better convergence with respect to the compared methods.  The article is dense but relatively fine to follow. Theoretical development seems to be complete and accurate, though I admit I have not throughly followed the full derivation. Experimental section is in accordance with the theoretical claims and is more than sufficient.  Just for the sake of reproducibility of the results an exhaustive pseudocode or repository should be made available as a companion to the article to further strength the autor's points.


Limited memory Kelley's Method Converges for Composite Convex and Submodular Objectives
The original simplicial method (OSM), a variant of the classic Kelley's cutting plane method, has been shown to converge to the minimizer of a composite convex and submodular objective, though no rate of convergence for this method was known. Moreover, OSM is required to solve subproblems in each iteration whose size grows linearly in the number of iterations. We propose a limited memory version of Kelley's method (L-KM) and of OSM that requires limited memory (at most n + 1 constraints for an n-dimensional problem) independent of the iteration. We prove convergence for L-KM when the convex part of the objective (g) is strongly convex and show it converges linearly when g is also smooth. Our analysis relies on duality between minimization of the composite objective and minimization of a convex function over the corresponding submodular base polytope. We introduce a limited memory version, L-FCFW, of the Fully-Corrective FrankWolfe (FCFW) method with approximate correction, to solve the dual problem. We show that L-FCFW and L-KM are dual algorithms that produce the same sequence of iterates; hence both converge linearly (when g is smooth and strongly convex) and with limited memory. We propose L-KM to minimize composite convex and submodular objectives; however, our results on L-FCFW hold for general polytopes and may be of independent interest.

Summary: ------- The paper proposes a simple modification of the Original Simplicial Method that keeps in memory only the active linear constraints. Then by linking the dual method to Fully corrective Frank Wolfe they get convergence rates.  Quality & Clarity: ------------------ The paper is well written and motivated for people in the field. Note that it is probably quite abstruse otherwise. Its place in the literature is well defined. Proofs are correct.  The final rates (Theorem 2 and 3) are quickly stated: what is the total cost of the algorithm, taking into account the computations of the subproblems ? (by taking a concrete example where the subproblems can be computed efficiently) This would enlighten the impact in this rate of the limited memory implementation in comparison to the original simplicial method.  In Theorem 3, first, \Delta^{(i+1)} should be \Delta^{(i)}. Then these results only state that the estimated gap is bounded by the true gap, I don't get the sub-linear rate from reading these equations.  Originality & Significance: --------------------------- The paper introduces two simple, though efficient, ideas to develop and analyze an alternative of the simplicial method (looking only at the active linear constraints of each subproblem and drawing convergence rates from the Frank-Wolfe literature). Relating the dual of the simplicial method to Franck-Wolfe method was already done by Bach [2]. I think that the present paper is therefore only incremental. Moreover the authors do not give compelling applications (this is due to the restricted application of the original simplicial method). In particular, they do not give concrete examples where the subproblems can easily be computed, that would help to understand the practical impact of the method.   Minor comments: --------------- - l139: no parenthesis around w - l140: put footnote for closed function here - l 227: Replace "Moreover" by "Therefore" - l 378: must be strictly contained *in* - As stated from the beginning, OSM is not Kelley's method, so why is the algorithm called Limited memory Kelley's method and Limited memory simplicial method ? This confusion might weaken its referencing in the literature, I think.   -------------------------------------- Answer to author's feedback -------------------------------------- Thank you for your detailed feedback. I revised my "incremental" comment. The problem was clearly asked before and the answer you provided is simple, clear and interesting.


Deep, complex, invertible networks for inversion of transmission effects in multimode optical fibres
We use complex-weighted, deep networks to invert the effects of multimode optical fibre distortion of a coherent input image. We generated experimental data based on collections of optical fibre responses to greyscale input images generated with coherent light, by measuring only image amplitude (not amplitude and phase as is typical) at the output of 1 m and 10 m long, 105 µm diameter multimode fibre. This data is made available as the Optical fibre inverse problem Benchmark collection. The experimental data is used to train complex-weighted models with a range of regularisation approaches. A unitary regularisation approach for complexweighted networks is proposed which performs well in robustly inverting the fibre transmission matrix, which is compatible with the physical theory. A benefit of the unitary constraint is that it allows us to learn a forward unitary model and analytically invert it to solve the inverse problem. We demonstrate this approach, and outline how it has the potential to improve performance by incorporating knowledge of the phase shift induced by the spatial light modulator.

I have read this paper and found it to be a solid contribution both to the field of optics and to NIPS.  With respect to its contribution to optics, I will admit that I have not worked in this field for over two decades, but it does seem that they are solving an interesting problem in a new way.  The removal of speckle from images seems to be a relevant problem in the field:    (2018) SAR Image Despeckling Using a Convolutional Neural Network Puyang Wang, Student Member, IEEE, He Zhang, Student Member, IEEE and Vishal M. Patel, Senior Member, IEEE https://arxiv.org/pdf/1706.00552.pdf  Most cited (2010) "General Bayesian estimation for speckle noise reduction in optical coherence tomography retinal imagery" Alexander Wong, Akshaya Mishra, Kostadinka Bizheva, David A. Clausi1  And I could not find a similar work in my search.    The authors contribute compelling examples in their supplemental materials, which lends credence to the claims that their method actually works.  The authors also promise to contribute a novel dataset to use for ML benchmarking on this type of problem.    In my opinion, this paper comprises what I believe a NIPS application paper should have, a paper that would be considered best in class in it's field of application and one that uses neural information processing methods to bring a substantial advance to the application domain.  Additionally the contribution of a benchmark dataset should further motivate research on this problem area.  The paper is well written and easy to follow.


Dual Policy Iteration
A novel class of Approximate Policy Iteration (API) algorithms have recently demonstrated impressive practical performance (e.g., ExIt [1], AlphaGo-Zero [2]). This new family of algorithms maintains, and alternately optimizes, two policies: a fast, reactive policy (e.g., a deep neural network) deployed at test time, and a slow, non-reactive policy (e.g., Tree Search), that can plan multiple steps ahead. The reactive policy is updated under supervision from the non-reactive policy, while the non-reactive policy is improved via guidance from the reactive policy. In this work we study this class of Dual Policy Iteration (DPI) strategy in an alternating optimization framework and provide a convergence analysis that extends existing API theory. We also develop a special instance of this framework which reduces the update of non-reactive policies to model-based optimal control using learned local models, and provides a theoretically sound way of unifying model-free and model-based RL approaches with unknown dynamics. We demonstrate the efficacy of our approach on various continuous control Markov Decision Processes.

Major Comments:  1) page 3: How do you set a parameter $\alpha$ and what effect it has? In other words, if $\eta^* = \argmin J(\eta)$ and $\eta_{\alpha}$ is the solution of Eq , what can you say about relation between $\eta^*$ and $\eta_{\alpha}$?  2) page 3: parameter $\delta$ in Eq 3. Could you please elaborate more on how are you going to choose? If it does depend on policy $\pi_n$, then we need to replace it by $\delta_n$ (i.e. it is changing from iteration to iteration). Also, apparently, this parameter will have an effect on the transition policy $\hat{P}$ that is found, and, hence, has effect to the policy $\eta_n$ evaluated at iteration $n$. Did you study this effect?  3) page 3: Theorem 3.1. I am confused with this result. Because the real transition $P$ is not available then we need to study the $J(\eta_n)$ evaluated with model $\hat{P}$ instead of $P$. But Theorem 3.1 studies the quality of policy $\eta_n$ with respect to real model $P$. Therefore, Theorem 3.1 doesn't quantify the quality of the tree search policy $\eta_n$.   4) page 4: Section 3.2. How do you define parameter $\beta$ in Eq 6. How effect does it have on the approximate solution $\pi_{n+1}$?  5) Due to PDL Lemma, for a given fast reactive policy $\pi_n$ the corresponding tree search policy $\eta_n$ actually doesn't depend on $\pi_n$. This is because policy $eta_n = argmin_{\eta} J(\eta)$, and $J(\eta)$ doesn’t depend on $pi_n$. The dependence appears when we impose trust region constraint given in terms of $\pi_n$.  My biggest issue with this paper is about the parameters $\alpha, \delta, \beta$ that was introduced as some form of relaxation but, at the same time, no effect of them were studied. (Please see my comment on Theorem 3.1 in 3) for better understanding).   Minor Comments: 1) page 3: Although the min-max framework considered in Section 3 is understandable, it would be good to elaborate on for readers.  Apart from the above major and minor comments, overall the paper appears to be written and structured well. Technically novel.


Robust Hypothesis Testing Using Wasserstein Uncertainty Sets
We develop a novel computationally efficient and general framework for robust hypothesis testing. The new framework features a new way to construct uncertainty sets under the null and the alternative distributions, which are sets centered around the empirical distribution defined via Wasserstein metric, thus our approach is data-driven and free of distributional assumptions. We develop a convex safe approximation of the minimax formulation and show that such approximation renders a nearly-optimal detector among the family of all possible tests. By exploiting the structure of the least favorable distribution, we also develop a tractable reformulation of such approximation, with complexity independent of the dimension of observation space and can be nearly sample-size-independent in general. Real-data example using human activity data demonstrated the excellent performance of the new robust detector.

The rebuttal addressed my technical concerns, and also I seemed to have misjudged the size of the contributions at first. My score has been updated.   This paper studies the two-sample non-parametric hypothesis testing problem. Given two collections of probability distribution, the paper studies approximating the best detector against the worst distributions from both collections. A standard surrogate loss approximation is used to upper bound the worst case risk (the maximum of the type I and type II errors) with a convex surrogate function, which is known to yield a good solution.   Some parts of section 3 are novel, but I'm not familiar enough with the literature to pinpoint exactly what. The convex surrogate approximation is a standard relaxation in learning theory. I wish the authors be more clear in this manner.   The main novel contribution seems to be in the setting where the two collections of distributions are wasserstein balls centered around empirical distributions. It that case, Theorem 3 derives a convex relaxation to solving the optimal detector.   The authors then provide experimental justification for their method, showing improved performance on real data from previous algorithms.   The writing was generally good, though I found the high level ideas hard to follow. Please be more clear about which results apply in which setting. Also, the authors really need to justify why considering wasserstein balls around empirical distributions is a good idea. In fact, I have a few technical concerns:  1. What if the two empirical distributions from P1 and P2 have little overlap? It seems  that the hypothesis testing problem becomes trivial  2. Is prop 1 only for the Wasserstein-ball sets? The proof certainly needs this assumption but it's not in the proposition statement.  3. Why in the objective for problem (6) concave? What assumptions on \phi are needed?  4. Please justify the min-max swap in 374   To summarize, given my technical concerns and the fact that the main contribution seems to be deriving a convex relaxation, I'm a bit hesitant to recommend this paper.


ρ-POMDPs have Lipschitz-Continuous -Optimal Value Functions
Many state-of-the-art algorithms for solving Partially Observable Markov Decision Processes (POMDPs) rely on turning the problem into a "fully observable" problem-a belief MDP-and exploiting the piece-wise linearity and convexity (PWLC) of the optimal value function in this new state space (the belief simplex ∆). This approach has been extended to solving ρ-POMDPs-i.e., for informationoriented criteria-when the reward ρ is convex in ∆. General ρ-POMDPs can also be turned into "fully observable" problems, but with no means to exploit the PWLC property. In this paper, we focus on POMDPs and ρ-POMDPs with λ ρ -Lipschitz reward function, and demonstrate that, for finite horizons, the optimal value function is Lipschitz-continuous. Then, value function approximators are proposed for both upper-and lower-bounding the optimal value function, which are shown to provide uniformly improvable bounds. This allows proposing two algorithms derived from HSVI which are empirically evaluated on various benchmark problems.

post-rebuttal: I have a small suggestion for the authors to include the related work in the paper if they can and to put a small experiment in the paper where surveillance is the goal of the agent. This is the main motivation of the paper. Overall I am happy with the paper and response.   ----- Summary: This paper presents a method for solving POMDPs (offline POMDP planning) when the value function of the POMDP is not necessarily convex (mainly breaking the PWLC property exploited by most POMDP planners). The paper presents a new POMDP planner based on heuristic search value iteration that does not rely on convexity of the optimal value function to compute it. The idea is to use upper and lower bound on the value function (and then tighten them at various beliefs) to compute the optimal value function (or an approximation of it). The property that the paper exploits to generate these upper and lower bounds is that the optimal value function has to be Lipschitz continuous. The paper also shows that if the immediate expected reward is Lipschitz continuous then the optimal value function is guaranteed to be Lipschitz continuous and then exploits this property of the value function to propose upper and lower bounds on the value function by obtaining an expression for the Lipschitz contant for the curves (cones in this case) to upper and lower bound the value function. Finally, the paper gives empirical results on standard POMDP problems in literature.   Quality - This is a good quality paper; thorough and rigorous for most parts.   Clarity: The paper is clear and well-written.   Originality: The paper presents original work.   Significance: I think the paper presents significant results that are relevant to this community.   Strength: I believe the main strength of the paper is the carefully designed and principle method for obtaining Lipschitz continuous upper and lower bound on the value function of a POMDP.   Weakness:  - I am quite not convinced by the experimental results of this paper. The paper sets to solve POMDP problem with non-convex value function. To motivate the case for their solution the examples of POMDP problem with non-convex value functions used are: (a) surveillance in museums with thresholded rewards; (b) privacy preserving data collection.  So then the first question is when the case we are trying to solve are above two, why is there not a single experiment on such a setting, not even a simulated one? This basically makes the experiments section not quite useful.   - How does the reader know that the reward definitions of rho for this tasks necessitates a non-convex reward function. Surveillance and data collection has been studied in POMDP context by many papers. Fortunately/unfortunately, many of these papers show that the increase in the reward due to a rho based PWLC reward in comparison to a corresponding PWLC state-based reward (R(s,a)) is not that big. (Papers from Mykel Kochenderfer, Matthijs Spaan, Shimon Whiteson are some I can remember from top of my head.) The related work section while missing from the paper, if existed, should cover papers from these groups, some on exactly the same topic (surveillance and data collection).   - This basically means that we have devised a new method for solving non-convex value function POMDPs, but do we really need to do all that work?  The current version of the paper does not answer this question to me. Also, follow up question would be exactly what situation do I want to use the methodology proposed by this paper vs the existing methods.   In terms of critisim of significance, the above points can be summarized as why should I care about this method when I do not see the results on problem the method is supposedly designed for.


Blockwise Parallel Decoding for Deep Autoregressive Models
Deep autoregressive sequence-to-sequence models have demonstrated impressive performance across a wide variety of tasks in recent years. While common architecture classes such as recurrent, convolutional, and self-attention networks make different trade-offs between the amount of computation needed per layer and the length of the critical path at training time, generation still remains an inherently sequential process. To overcome this limitation, we propose a novel blockwise parallel decoding scheme in which we make predictions for multiple time steps in parallel then back off to the longest prefix validated by a scoring model. This allows for substantial theoretical improvements in generation speed when applied to architectures that can process output sequences in parallel. We verify our approach empirically through a series of experiments using state-of-the-art self-attention models for machine translation and image super-resolution, achieving iteration reductions of up to 2x over a baseline greedy decoder with no loss in quality, or up to 7x in exchange for a slight decrease in performance. In terms of wall-clock time, our fastest models exhibit real-time speedups of up to 4x over standard greedy decoding.

This paper presents a parallel inference algorithm that allows performing greedy decoding in sub-linear time. The proposed approach considers using k auxiliary models, in which each one predicts the j-th next word. In this way, these k models can run and predict the next k words in parallel. Then a verification step is conducted to verify the prediction of these k models. If the proposal is correct, then the predictions are accepted and the decoding can move on. In this way, in the best case, the model only requires m/k step to generate a sequence with m tokens.    Pros:  - The idea proposed in this paper is neat and interesting.  - The authors present experiments on tasks in two different domains to empirically demonstrate the efficiency of the decoding algorithm. - The authors proposed both exact and inexact algorithms for making the predictions. - The authors provide a nice analysis of their approach.   Cons:  - The writing of the paper can be improved, and the description of the proposed approach is sometimes confusing and descriptions of experimental settings and results are handwaving.  - The speed-up of the decoding is not guaranteed and require large duplicated computation, although the computation can be amortized.    Comments:  - It is unclear what the knowledge distillation approach is actually used in the experiment.  - I do not fully understand Table 1. It is unclear if the results showing is the exact or the approximate inference approach. It seems that the results showing here is using the exact greedy approach as the authors called it regular. But, in this case, why the Blue score is different for different k? If I understand correctly the p_1 model for different k should be the same, so the exact decoding should give the same output.  - Line 210-214: for approaches except "Regular", the authors should quantify the quality of the decoding quantitively. For example, what is the mean difference in pixel values between images generated by approximation greedy decoding and exact greedy decoding? How many of them can the difference be perceived by a human?     Minor comments:  - Figure 2 is not readable when the paper is printed in white and black.  - Line 109: Figure 3 is not showing what the authors describe.  - Line 199-120: Please explicitly define what do you mean by "Regular" and "Approximate"  in Table 2.   ----- Comments after rebuttal:  Thanks for the clarification and additional experiments on showing performance measured under wall-clock time and Table 1. With the clarification and the results on wall-clock, I update my rating.


Spectral Filtering for General Linear Dynamical Systems
We give a polynomial-time algorithm for learning latent-state linear dynamical systems without system identification, and without assumptions on the spectral radius of the system's transition matrix. The algorithm extends the recently introduced technique of spectral filtering, previously applied only to systems with a symmetric transition matrix, using a novel convex relaxation to allow for the efficient identification of phases.

Summary ------- This paper presents an algorithm for predicting the (one-step ahead) output of an unknown linear time-invariant dynamical system (LDS). They key benefits of the proposed algorithm are as follows: - the regret (difference between the prediction error of the algorithm, and the prediction error of best LDS) is bounded by sqrt(T) (ignoring logarithmic factors), where T is the length of the data sequence over which predictions are made. - the regret bound does not depend on the spectral radius of the transition matrix of the system (in the case that data is generated by a LDS). - the algorithm runtime is polynomial in the 'natural parameters' of the problem.  The algorithm is based on a recently proposed 'spectral filtering' technique [HSZ17]. In previous work, this method was only applicable to systems with a symmetric transition matrix (i.e. real poles). The present paper extends the method to general LDSs (i.e. non-symmetric transition matrices, or equivalently, complex poles).  The performance of the proposed algorithm is illustrated via numerical simulations, and is compared with popular existing methods including expectation maximization (EM) and subspace identification.   Quality ------- In my view, the paper is of very high quality. The technical claims are well-substantiated by thorough proofs in the supplementary material, and the numerical experiments (though brief) are compelling and relevant (i.e. comparisons are made to methods that are routinely used in practice).   Clarity ------- In my view, the paper is well-written and does a good job of balancing technical rigor with higher-level explanations of the key concepts. Overall, it is quite clear. I do have a few comments/queries:  In my opinion, the supplementary material is quite crucial to the paper; not just in the obvious sense (the supplementary material contains the proofs of the technical results), but personally, I found some of the sections quite difficult to follow without first reading the supplementary material, e.g. Section 4.1. I very much like the idea of giving a high-level overview and providing intuition for a result/proof, but to be honest, I found it quite difficult to follow these high-level discussion without first reading through the proofs. This is probably the result of a combination of a few factors: i) the results are quite technical, ii) these ideas (spectral-filtering) are quite new/not yet well known, iii) my own limitations. This is not necessarily a problem, as the supplementary material is well written.  Regarding the presentation of some of the technical details:  Should \Theta be \hat{\Theta} in Line 188 and equation (5)?  In the supplementary material, the reference to (equation?) (2) doesn't seem to make sense, e.g., line 33 but also elsewhere. Do you mean Definition 2?  In Line 18, do you mean \sum_{j=0}^\tau\beta_jA^{i-j} = A^{i-\tau}p(A) = 0?  In equation (10), is there an extra C?  It's not a big deal, but i is used as both sqrt(-1) (e.g. (29)) and as an index, which can be a little confusing at first.  Below (28), should M'_\omega be M'_l? Furthermore, should M'_l be complex, not real?  If M'_l is real, then where does (31) come from? (I would understand if M'_l was complex).  Are the r's in (38) and below missing an l subscript?     Concerning reproducibility of numerical results:  I would imagine that the numerical results are a little difficult to reproduce. For instance, the details on the EM algorithm (initialization method?) and SSID (number of lags used in the Hankel matrix?) are not given.   Perhaps Line 174 'the parameter choices that give the best asymptotic theoretical guarantees are given in the appendix' could be a bit more specific (the appendices are quite large). Also, Unless I missed it, in Sec 5 k is not specified, nor is it clear why W = 100 is chosen.    Originality ----------- The paper builds upon the previous work of [HSZ17] which considers the same one-step ahead prediction problem for LDS. The novelty is in extending this approach to a more general setting; specifically, handling the case of linear systems with asymmetric state transition matrices. In my view, this represents a sufficiently novel/original contribution.   Significance  ------------ The problem of time-series prediction is highly relevant to a number of fields and application areas, and the paper does a good job of motivating this in Section 1.    Minor corrections & comments ----------------------------  Full stop after the equation between lines 32-33.  Line 39, 'assumed to be [proportional to] a small constant'?  It's a little strange to put Algorithm 1 and Theorem 1 before Definition 2, given that Definition 2 is necessary to understand the algorithm (and to some extent the theorem).   Line 185 'Consider a noiseless LDS'.