Accelerated Mini-batch Randomized Block Coordinate Descent Method
We consider regularized empirical risk minimization problems. In particular, we minimize the sum of a smooth empirical risk function and a nonsmooth regularization function. When the regularization function is block separable, we can solve the minimization problems in a randomized block coordinate descent (RBCD) manner. Existing RBCD methods usually decrease the objective value by exploiting the partial gradient of a randomly selected block of coordinates in each iteration. Thus they need all data to be accessible so that the partial gradient of the block gradient can be exactly obtained. However, such a "batch" setting may be computationally expensive in practice. In this paper, we propose a mini-batch randomized block coordinate descent (MRBCD) method, which estimates the partial gradient of the selected block based on a mini-batch of randomly sampled data in each iteration. We further accelerate the MRBCD method by exploiting the semi-stochastic optimization scheme, which effectively reduces the variance of the partial gradient estimators. Theoretically, we show that for strongly convex functions, the MRBCD method attains lower overall iteration complexity than existing RBCD methods. As an application, we further trim the MRBCD method to solve the regularized sparse learning problems. Our numerical experiments shows that the MRBCD method naturally exploits the sparsity structure and achieves better computational performance than existing methods.
This paper explores the integration of mini-batch, randomized block CCD, and variance reduction for regularized empirical risk minimization problem. The major two problems the author target is variance reduction and the incremental proximal point problems when dealing with empirical and expected risk function. This method aims to show advantage over a recently proposed method, SPVRG.Quality:The experiments seem not enough. There are a couple of recently proposed methods aiming the same problem the paper cited, such as semi-SG [6], SVRG [5] and (SAG) [14]. The reviewer would like to see the comparison results of these methods. The paper compared with BPG and BRBCD, which are batch methods and thus the comparison is not fair enough. Besides the lack of comparisons, the current experimental results are not so exciting, compared with the recently proposed SPVRG algorithm which also aiming at shooting the variance reduction problem, the proposed MRBCD-II shows visible yet minor improvement on two moderate-size domains. Experiments on larger domains, if applicable, is preferred.Clarity:Some details regarding the paper：The paper is not quite well written, there are some typos which can be easily found, e.g.,line 160 'shows how to reduce shows how to reduce'line 217 'full take' should be 'fully take'line 193-194, 'These adjustments can guarantee ... goes to zero'. Please give proof on this.In experiment 1, from  line 340-344, different stepsizes are chosen according to different approaches. The reviewer wants to know how they are chosen and if they are optimal stepsize w.r.t the method to make a fair comparison.The paper goes without a conclusion part, which is fine, yet it is better to be with a conclusion part. Originality:The contribution seems incremental.Significance:The paper could be improved given more detailed analysis along with comprehensive comparison experimental study and result analysis.