Semi-Proximal Mirror-Prox for Nonsmooth Composite Minimization
We propose a new first-order optimization algorithm to solve high-dimensional non-smooth composite minimization problems. Typical examples of such problems have an objective that decomposes into a non-smooth empirical risk part and a non-smooth regularization penalty. The proposed algorithm, called SemiProximal Mirror-Prox, leverages the saddle point representation of one part of the objective while handling the other part of the objective via linear minimization over the domain. The algorithm stands in contrast with more classical proximal gradient algorithms with smoothing, which require the computation of proximal operators at each iteration and can therefore be impractical for high-dimensional problems. We establish the theoretical convergence rate of Semi-Proximal MirrorProx, which exhibits the optimal complexity bounds, i.e. O(1/ǫ 2 ), for the number of calls to linear minimization oracle. We present promising experimental results showing the interest of the approach in comparison to competing methods.
After the rebuttal:The theory developed for inexact composite mirror prox is a trivial modification of the proof of Theorem 3.1 in [10]. Authors do not hide this fact. In the end, idea-wise this paper is close to [14] and technically it does not appear to be too much involved. To me, the most interesting point of this work is that the objective function can be doubly non-smooth, in the sense f can also be non-smooth. On that front, how does this work relate tohttp://arxiv.org/abs/1502.03123A. Yurtsever, Q. Tran-Dinh, and V. Cevher "Universal Primal-Dual Proximal-Gradient Methods"which appears also to solve non-smooth constrained problems with Fenchel-type oracles?  ============ This paper proposes a new first order optimization algorithm to solve the composite problem (1) under assumptions A1-A4,S1-S3 (see the paper), which can be considered as a variant of composite mirror prox method (CMP), where the prox-operator are solved inexactly using composite conditional gradient algorithm (CCG). This paper lacks the references for some related recent works (e.g. [1] and [2]). Technically, the authors show how the inexactness in the CMP iterates accumulate in the dual gap function, they explain how we should choose the accuracy input for CCG subroutines and provide the corresponding worst case complexity analysis, which can be considered as the novel part of the work.However, from an intellectual point of view, the idea of solving prox-operator approximately reminds us [2] and it is not clear how CCG subroutine that is presented in the paper is any different from the algorithms presented in [1]. (Note that the assumption (13) is a consequence of the Holder continuity assumption in [1]).Considering the presence of [1] and [2], the significance of this work becomes questionable.  Finally, we provide below a list of minor issues and typos for the authors: 1. US English and UK English are used inconsistently. In some parts of the text 'minimization' and 'optimization' are used whereas in some other parts these are written as 'minimisation' and 'optimisation'. 2. line 91 - point is missing at line after reference [13]. 3. line 141 - 'be' missing: can 'be' much cheaper. 4. line 214 - 'of' missing: in terms 'of' this dual gap function ... 5. line 217 - typo : fro - for 6. line 247 - there is an extra parenthesis ')' at the end of line. 7. line 256 - typo: proc-mapping - prox-mapping 8. line 271 - L < inf should be L_0 < inf / or L_0 in eq (13) should be L 9. line 278 - typo: given 'on' input - given 'an' input 10. line 324 - repetition: we assume that 'we we' have 11. line 365 - Assumptions (B.1)-(B.3) should be replaced by Assumptions (S.1)-(S.3) 12. line 776 - We shall compare and 'compare' - We shall compare and 'contrast' [1] Y. Nesterov, Complexity Bounds for Primal-dual Methods Minimizing the Model of Objective Function, CORE Discussion Paper, 2015. [2] G. Lan, Gradient Sliding for Composite Optimization, arxiv:1406.0919v2.