Multi-Step Stochastic ADMM in High Dimensions: Applications to Sparse Optimization and Matrix Decomposition
In this paper, we consider a multi-step version of the stochastic ADMM method with efficient guarantees for high-dimensional problems. We first analyze the simple setting, where the optimization problem consists of a loss function and a single regularizer (e.g. sparse optimization), and then extend to the multi-block setting with multiple regularizers and multiple variables (e.g. matrix decomposition into sparse and low rank components). For the sparse optimization problem, our method achieves the minimax rate of O(s log d/T ) for s-sparse problems in d dimensions in T steps, and is thus, unimprovable by any method up to constant factors. For the matrix decomposition problem with a general loss function, we analyze the multi-step ADMM with multiple blocks. We establish O(1/T ) rate and efficient scaling as the size of matrix grows. For natural noise models (e.g. independent noise), our convergence rate is minimax-optimal. Thus, we establish tight convergence guarantees for multi-block ADMM in high dimensions. Experiments show that for both sparse optimization and matrix decomposition problems, our algorithm outperforms the state-of-the-art methods.
This paper proposed a multistep version of Stochastic ADMM method for high-dimension problems, in which there are multiple epochs defined. In each epoch, projections onto certain norm balls with decreasing radius are employed.As to the theoretical results, the authors showed that this simple variant achieves the minimax bound for sparse regression problem; while considering matrix decomposition problem, an excess risk bound, which is worse by a factor of d than the batch version, is obtained. Numerical simulation confirmed that the proposed algorithm performs better under both cases.The work is a little incremental considering that the proposed method is the combination of ST-ADMM with the method proposed in [5].  Both algorithms have the same algorithmic framework. In fact, the O(s \log d /T) excess risk bound is not very surprising, because O(1/T) comes from the doubling trick used in [5] and an earlier work [**][**] Elad Hazan, Satyen Kale, Beyond the regret minimization barrier: an optimal algorithm for stochastic strongly-convex optimization, COLT’11The factor s\logd is standard for high-dimensional problem with sparse structure. Furthermore, the authors mistakenly use regret bound to refer to excess risk bound through the paper, which is misleading.More surprisingly, the proposed method using stochastic ADMM outperforms [5] on sparse regression significantly in the empirical experiments, while they have the same order of excess rick bound. Note that sparse regression is a non-constrained problem in nature. Solving it using ADMM by introducing some auxiliary variable works better than directly solving the unconstrained problem using stochastic RDA is very counter-intuitive.Also, the organization of this paper should also be improved for better understanding. It is better to present a unified algorithm, and then give two examples, rather than presenting two algorithms separately for sparse regression and matrix decomposition.====The authors explained why their algorithm outperforms multi-step RDA in the rebuttal. I have no concern about the empirical experiments any more.