SEGA: Variance Reduction via Gradient Sketching
We propose a randomized first order optimization method-SEGA (SkEtched GrAdient)-which progressively throughout its iterations builds a variancereduced estimate of the gradient from random linear measurements (sketches) of the gradient. In each iteration, SEGA updates the current estimate of the gradient through a sketch-and-project operation using the information provided by the latest sketch, and this is subsequently used to compute an unbiased estimate of the true gradient through a random relaxation procedure. This unbiased estimate is then used to perform a gradient step. Unlike standard subspace descent methods, such as coordinate descent, SEGA can be used for optimization problems with a non-separable proximal term. We provide a general convergence analysis and prove linear convergence for strongly convex objectives. In the special case of coordinate sketches, SEGA can be enhanced with various techniques such as importance sampling, minibatching and acceleration, and its rate is up to a small constant factor identical to the best-known rate of coordinate descent.
In this paper, the authors propose a randomized first order optimization method (SEGA) which progressively builds a variance reduced estimate of the gradient from random linear measurements of the gradient. The proposed method (or class of methods - depending on the sketch matrix and metric used) updates the current estimate of the gradient through a sketch-and-project operation using new gradient information and the past estimate of the gradient.  - The first half of the paper is very well written, easy to follow and well motivated. However, the quality of the paper deteriorates after page 6. The paper has minor typos and grammatical mistakes that can be corrected easily.  — Line 29: now draw a two simple -> now draw two simple — Line 36: methods as stochastic gradient -> methods such as stochastic gradient — Line 83: For the illustration, -> To illustrate this, — Line 83: shows a the evolution -> shows the evolution — Line 87: baseline choice -> baseline choices — Line 106: no proximal setting, coordinate -> no proximal term and coordinate — Line 115: eigen-properties or matrices -> eigen-properties of matrices — Line 148: analogy -> analog (and other instances of this) — Line 152: fresult -> result — Line 158: ary -> Corollary — Line 165: in sense -> in the sense — Line 181: verify claim -> verify the claim — Line 192: a construct -> a constructed problem — Line 193: system comparing -> system compared  - The Experiments section seems weak, primarily because all problems solved are constructed problems. The experiments are well though out to highlight certain algorithmic features of the method, however, several details are missing (e.g., what is the dimension n of the problems solved?), comparison with more methods would strengthen the claims made and experiments on real ML problems would highlight the merits (and limitations) of SEGA. — Section 5.1: —— The authors should mention in what settings different cost scenarios arise for solving the linear systems. —— The authors should compare the methods on problems that are not quadratic. — Section 5.2: —— How was the finite difference interval set for the gradient estimation used in SEGA? Did the authors use forward differences (n function evaluations per iteration)?  ——The authors should compare against other zero order optimization methods (e.g., model based trust region method, finite difference quasi-Newton methods or Nelder-Mead). —— Again, the authors should compare the methods on problems that are not quadratic. —— In this section do the authors assume that they have access to true (non-noisy) function evaluations? — Section 5.3: —— How is beta chosen in the accelerated variant of the method? —— In practice, on real world problems, how would one chose the subspaces to use in subspaceSEGA? The authors should make a comment about this in the paper. — The authors claim that “theory supported step sizes were chosen for all experiments.” These step sizes depend on problem specific parameters such as Trace(M), minimum and maximum eigenvalues of different matrices and other quantities. How were these quantities calculated?   - Other minor issues and questions: — The authors should mention clearly that it is not necessary to compute the full gradient in Algorithm 1. — “SEGA can be seen as a variance reduced coordinate descent.” Is it not the case that SEGA is more general than that? If certain sketch matrices are chosen then it is a coordinate descent type method, but with other sketches this is not the case.  — “potentially more expensive iteration”: it would be helpful for the reader if the authors provided some examples. — “total complexity”: do the authors mean iteration complexity? — What is alpha_0 in Corollary 3.4? — Line 128: “Corollary 4.3 and Corollary 4.3”? — Remark 4.4: “Therefore, SEGA also works in a non-convex setting.” I believe this statement is too strong. SEGA works for a certain class of non-convex function that satisfy the PL inequality (these function have unique minimizers). — Corollary 4.5: Is the Lyapunov function used here the same as in Theorem 4.2? — Table 1: What is the cause of the constant differences in the results of CD and SEGA? Is this an artifact of the analysis? Is this realized in practice? — The authors should consider adding a Final Remarks section to summarize their contributions and to aid the reader.  - I believe the generality of the algorithms the authors propose (different sketch matrices) and the special cases of algorithms that they recover, as well as the wide range of potential problem classes that can be solved by the proposed methods, makes this a paper of interest for the ML and Optimization communities. However, there are certain issues and questions with the paper (mentioned above) that should be addressed. As such, I marginally recommend this paper for publication at NIPS.  I have read the author rebuttal and stand by my initial assessment of the paper.