Better Approximation and Faster Algorithm Using the Proximal Average
It is a common practice to approximate "complicated" functions with more friendly ones.
In large-scale machine learning applications, nonsmooth losses/regularizers that entail great computational challenges are usually approximated by smooth functions. We re-examine this powerful methodology and point out a nonsmooth approximation which simply pretends the linearity of the proximal map. The new approximation is justified using a recent convex analysis toolproximal average, and yields a novel proximal gradient algorithm that is strictly better than the one based on smoothing, without incurring any extra overhead. Numerical experiments conducted on two important applications, overlapping group lasso and graph-guided fused lasso, corroborate the theoretical claims.
This paper describes a new proximal-gradient based 
      procedure for the minimization of an objective function, composed by the 
      sum of a convex smooth term and several nonsmooth convex (regularization) 
      terms. The motivation comes from the fact that often, and in significant 
      cases, proximal gradient schemes [e.g. FISTA] are not directly applicable 
      to that scenario, since the proximity operator of the sum of several 
      convex functions is not explicitly computable from the proximity operator 
      of each individual term. The proposed algorithm deals directly with these 
      individual prox operators. The method is actually a mix of 
      approximation techniques (smoothing) as developed by Nesterov [5] and 
      proximal gradient algorithms. Indeed they propose to replace the whole set 
      of nonsmooth terms, with a unique nonsmooth (close) approximation for 
      which the proximity operator is computable; and this approximation turns 
      out to be the proximal average of the regularization terms --- which has 
      bees studied in detail in [11]. Solutions of the approximate problem are 
      then related to the solutions of the original one and a complexity of 
      $O(1/\varepsilon)$ is shown. The authors claim this procedure to be faster 
      then the one based on the smoothing technique (S-APG algorithm [13]). 
      Indeed, although the asymptotic behavior remains the same, an improvement 
      of the constants is proved. This improvement is finally shown in the 
      experimental section. I find the proposed method quite interesting 
      and the main idea, of approximate with a nicer nonsmooth term instead of 
      smoothing, valuable and original. A weak aspect is that the nonsmooth 
      terms are required to be (globally) Lipschitz continuous, hence finite 
      everywhere. This means that constraints cannot be treated. The theoretical 
      part is generally sound and precise. Unfortunately I am afraid that the 
      proof of the main theorem is not fully correct. Nevertheless the statement 
      is correct and the problem in the proof can be overcome without much 
      effort as I show below. As regards the experimental section, perhaps other 
      comparisons with more recent approaches (like the two loop-algorithm 
      proposed in [10]) could have been useful. Anyway it is clearly stated that 
      the algorithm is meant to improve S-APG and the section of experiments 
      addresses this comparison. TYPOS, SMALL CORRECTIONS AND 
      SUGGESTIONS: -- In the properties listed in Prop. 1, I would add 
      the differentiability of the Moreau envelope and the explicit expression 
      of the gradient $\nabla M_f^\mu = \mu^{-1} (Id - P_f^\mu)$. -- If 
      $SS_\mu$ is the class of $\mu$-Lipschitz continuous functions, then $f \in 
      SC_\mu \iff f^* \in SS_{1/\mu}$ (not $f^* \in SS_\mu$) and $M^\mu : 
      \Gamma_0 \to SS_{1/\mu}$. Moreover $M^\mu : \Gamma_0 \to SS_{1/\mu}$ is 
      concave only on finite valued functions, not on the whole space 
      $\Gamma_0$, since the convex combination of two functions in $\Gamma_0$ 
      might be even not proper. Thus, Prop. 2 should be accordingly corrected. 
      -- All the results between Def 1 and Prop 3 are in [11]. This 
      should be clearly stated. A CRITICAL POINT: I think 
      that the proof of Theorem 1 is not quite correct and should be pursued 
      with more care. The criticism relies on the fact that generally in prox 
      gradient algorithms, the bound on the precision (after a given number of 
      iterations) depends on the distance between the starting point and the 
      solution set of the problem. This means that the notation $O(\sqrt{\frac 1 
      {\mu \varepsilon}})$, recalled in the proof, actually hides a further 
      dependence on $\mu$ in the constant --- since it is the approximate 
      function that is being minimized. To overcome this problem, I suggest 
      the authors to follow the path of [13] (see Proof of Theorem 2 in the 
      appendix). Indeed if one chooses $\mu \leq 1/L_0$ and minimizes the 
      approximation $F_\mu(x) = \gamma \ell(x) + A^\mu(x)$ with FISTA, then for 
      every $x \in H$ it holds $$ F_\mu(x_t) - F_\mu(x) \leq 2 \frac 1 
      \mu \frac {\Vert x_0 - x \Vert^2}{t^2}\,. $$ Note that this 
      inequality is valid for every $x \in H$, not only for the minimizers of 
      $F_\mu$ (just look at the proof of FISTA in [2] or [10]). Then if we now 
      take $x_*$ as a minimizer of the original function $F$ and plug it into 
      the previous inequality we end up with $$ F(x_t) - F(x_*) \leq 
      \frac{\mu \overline{L^2}}{2} + 2\frac{1}{\mu} \frac {\Vert x_0 - x_* 
      \Vert^2}{t^2}. $$ This way the constant term $\Vert x_0 - x_* 
      \Vert^2$ does not depend anymore from the approximate problem (hence from 
      $\mu$), but only on the original problem (see also the discussion in 
      section 3.6 in [13]). The statement of the theorem remains unchanged.