Estimation, Optimization, and Parallelism when Data is Sparse
We study stochastic optimization problems when the data is sparse, which is in a sense dual to current perspectives on high-dimensional statistical learning and optimization. We highlight both the difficulties-in terms of increased sample complexity that sparse data necessitates-and the potential benefits, in terms of allowing parallelism and asynchrony in the design of algorithms. Concretely, we derive matching upper and lower bounds on the minimax rate for optimization and learning with sparse data, and we exhibit algorithms achieving these rates. We also show how leveraging sparsity leads to (still minimax optimal) parallel and asynchronous algorithms, providing experimental evidence complementing our theoretical results on several medium to large-scale learning tasks.
This paper studies the effect of data sparsity on 
      learning and optimization. Specifically, the paper assumes that the 
      support of gradient of an objective function is in the support of feature 
      vectors, where feature vectors are considered to be sparse. Definding 
      appearance probabablities on feature, the paper derives a lower bound on 
      the gap between an estimated solution from minimization with given data 
      and a true minimizer, and shows that ADAGRAD methods [4,8] achieves this 
      lower bound (and thereby optimal). The paper also suggests an algorithm 
      using asynchronous updates, showing that the suggested algorithm achieves 
      the optimal bound up to some factors depending on log(dimensionality) and 
      update delay probabilities. Quality: The claims are supported 
      by proved theorems. Although the claims are technically sound, I believe 
      the main assumption in Eq.(2) excludes many popular approaches with 
      regularization, for example L1 (lasso), L2 (ridge), group lasso, etc. The 
      weakness is indeed revealed in the experiments, for instance the 
      right-hand side plots in Figure 2. The plots shows the sensitivity of the 
      suggested algorithm to the choices of the steplength. In the experiments 
      the authors chose "optimal" steplengths by using test sets, but this 
      doesn't sound very practical. Also, it is assumed that the 
      distance from independence is bounded by some constant m, that is, 
      di(\rho) <= m, but I believe, depending on systems, it could be 
      proportional to the number of parallel workers k. If it is the case, then 
      the linear speedup using multiple cores in the end of Section 3.2 is 
      unlikely. The machines used in the experiment had only a limited number of 
      cores, so the validity of assumption was not very well supported. 
      Clarity: In general the arguments are clearly stated and well 
      organized. But some minor points can be improved for better readability: 
      - page 2, line 104: consist distributions -> consist of 
      distributions - page 3, Eq (6) and (7): It would be kind to note the 
      fact that R_2 ~ O(\sqrt(d)) for less-trained eyes. - page 3, the last 
      line: exponential improvement in the dimension: it is not clear to me why 
      "exponential" in the dimension. - page 5, line 236: Assuming the data 
      appearance... -> Assuming the feature appearance Originality: 
      I believe the introduction of the assumption in Eq.(2), the lower 
      bound in Proposition 1, the introduction of feature appearance probability 
      in this context, are new. The analysis for Hogwild algorithm seems to be 
      new as well, but I didn't go through the proof of that part. 
      Significance: The new analysis supporting asynchronous update 
      when features are sparse, will be an interesting message to the community. 
      However as the assumption in Eq.(2) excludes many interesting problems, I 
      don't think it will have a large impact.