Adaptive Primal-Dual Splitting Methods for Statistical Learning and Image Processing
The alternating direction method of multipliers (ADMM) is an important tool for solving complex optimization problems, but it involves minimization sub-steps that are often difficult to solve efficiently. The Primal-Dual Hybrid Gradient (PDHG) method is a powerful alternative that often has simpler sub-steps than ADMM, thus producing lower complexity solvers. Despite the flexibility of this method, PDHG is often impractical because it requires the careful choice of multiple stepsize parameters. There is often no intuitive way to choose these parameters to maximize efficiency, or even achieve convergence. We propose self-adaptive stepsize rules that automatically tune PDHG parameters for optimal convergence. We rigorously analyze our methods, and identify convergence rates. Numerical experiments show that adaptive PDHG has strong advantages over non-adaptive methods in terms of both efficiency and simplicity for the user.
Augmented Lagrangian algorithms like ADMM have become popular recently since they decouple an objective function consisting of the sum of several functionals. PDHG is a version were the subproblems have a particularly simple form, more specifically, they are proximal problems w.r.t. the individual functionals of the objective function. In this paper, the authors propose an adaptive step selection rule for PDHG, provide convergence guarantees and show the numerical performance of the new algorithm.The proposed algorithm seems to converge quickly according to the numerical experiments. However, there exist several PDHG algorithms which also use dynamic step sizes and have shown to outperform the methods used for comparison in the present paper. There is Algorithm 2 of [5] and the algorithm in "Adaptive Primal-Dual Hybrid Gradient Methods for Saddle-Point Problems" by T. Goldstein et al. (2013). The latter is particularly relevant here since it uses a very similar idea namely to balance the primal and dual residual. Without these comparisons it is a bit hard to judge how good the proposed algorithm actually is. A minor issue in this context is the question whether there are better constant step sizes than the two used in the experiments.Another very relevant paper which is not cited here is "On the convergence of primal-dual hybrid gradient algorithms for total variation image restoration" by S. Bonettini and V. Ruggiero (2012) since it shows convergence of PDHG for dynamic step sizes.Moreover, it would be useful to give a brief explanation of the result of Theorem 1. For example, one could clearly state that the nominator is bounded from above since u^k is bounded (Lemma 2). Also, providing the convergence result in terms of an average over {u^1,...,u^t} seems a bit unusual. Related to this is the question whether the sequences x^k or y^k converge. Minor issues: * It should be \tilde{u}^T in Theorem 1. * C_\phi in Lemma 3 should be defined before it is used. * In (20), \delta_k-1 is not defined. One could directly replace (1-\delta_k-1) with \phi_k-1