Non-Ergodic Alternating Proximal Augmented Lagrangian Algorithms with Optimal Rates
We develop two new non-ergodic alternating proximal augmented Lagrangian algorithms (NEAPAL) to solve a class of nonsmooth constrained convex optimization problems. Our approach relies on a novel combination of the augmented Lagrangian framework, alternating/linearization scheme, Nesterov's acceleration techniques, and adaptive strategy for parameters. Our algorithms have several new features compared to existing methods. Firstly, they have a Nesterov's acceleration step on the primal variables compared to the dual one in several methods in the literature. Secondly, they achieve non-ergodic optimal convergence rates under standard assumptions, i.e. an O 1 k rate without any smoothness or strong convexity-type assumption, or an O 1 k 2 rate under only semi-strong convexity, where k is the iteration counter. Thirdly, they preserve or have better per-iteration complexity compared to existing algorithms. Fourthly, they can be implemented in a parallel fashion. Finally, all the parameters are adaptively updated without heuristic tuning. We verify our algorithms on different numerical examples and compare them with some state-of-the-art methods.
Summary:   The paper proposed 4 new variants of Augmented Lagrangian methods, which called NAPALA (non-ergodic alternating proximal augmented Lagrangian algorithms) to solve non-smooth constrained convex optimization problems under different problem structure assumptions. The first algorithm only requires f and g to be convex but neither necessarily strongly convex nor smooth. Especially, its parallel variant (8) has more advantages. The second algorithm requires either f or g to be strongly convex. Its variant (12) also allows to have parallel steps. The author(s) could derive the convergence rate at the last iterate (non-ergodic), which are the state-of-the-art results. Numerical experiments also show good overall performance of the proposed algorithms compared to other algorithms.   Comments:   In general, I really like the results of the paper since it could achieve the convergence rate at the last iterate (non-ergodic sense), which is very useful for sparse and low rank optimization or image processing. In the non-strongly convex case, there exist many algorithms that can achieve O(1/k) convergence rate, but such a rate is often in an ergodic sense. Note that ergodic sequences possibly break the sparsity and low rankness in sparse and low-rank optimization, respectively. Especially, in image processing, ergodic sequences may destroy the sharp edge structure of images. Although the convergence rate of ADMM (O(1/k)) and its accelerated versions is widely studied in the literatures, this paper proposes 4 novel algorithms which I have not seen in the literature. Compared to existing methods in the literature, the parallel variant (8) seems to have several advantages while achieving O(1/k)-non-ergodic optimal rate under only convexity and strong duality. Algorithm 2 allows using different sequence to achieve either ergodic or nonergodic rate on y, but nonergodic rate on x. It also linearizes the y-subproblem compared to ADMM. I believe that the results of the paper are non-trivial and have significant theoretical contribution to optimization and machine learning community.   Besides the above mentioned positive points, I also have the following comments:   1) I saw that you are able to achieve the convergence rate of O(1/k^2) in the case that either f or g is strongly convex (section 4.1). However, when you do parallel step, you need f and g are strongly convex (together) to have the rate O(1/k^2). What do you think about the convergence rate when either f or g is strongly convex in this parallel step? Do you think it is still possible to achieve O(1/k^2)? Have you tried to do experiments to see the behavior?  2) Could you please explain more clearly about Section 4.3? What should we do if f and/or g are smooth?   3) A minor comment: I think your algorithms could also expand to coordinate descent. What is your opinion about this?