Optimal Algorithms for Non-Smooth Distributed Optimization in Networks
In this work, we consider the distributed optimization of non-smooth convex functions using a network of computing units. We investigate this problem under two regularity assumptions: (1) the Lipschitz continuity of the global objective function, and (2) the Lipschitz continuity of local individual functions. Under the local regularity assumption, we provide the first optimal first-order decentralized algorithm called multi-step primal-dual (MSPD) and its corresponding optimal convergence rate. A notable aspect of this result is that, for non-smooth functions, while the dominant term of the error is in O(1/ √ t), the structure of the communication network only impacts a second-order term in O(1/t), where t is time. In other words, the error due to limits in communication resources decreases at a fast rate even in the case of non-strongly-convex objective functions. Under the global regularity assumption, we provide a simple yet efficient algorithm called distributed randomized smoothing (DRS) based on a local smoothing of the objective function, and show that DRS is within a d 1/4 multiplicative factor of the optimal convergence rate, where d is the underlying dimension.
The paper studies distributed algorithms for non-smooth convex optimization. The authors consider a standard distributed optimization setting in which given a communication graph, where each node has access to its own non-smooth convex function, the goal is to globally minimize the average of functions in little as possible time, where time accounts for both computation on each node and communication between neighboring nodes in the graph. The authors consider two natural assumptions, 1) where the global function (i.e., average of local functions) is Lipschitz continuous and 2) when each local function is Lipschitz continuous. They also consider both distributed master/slave-type algorithms and completely decentralized gossip-based algorithms.  In the centralized master/slave settings, the authors show that under the global Lipschitz assumption, a distributed version of Nesterov's accelerated gradient method (slave nodes only compute gradients and send to master node which runs Nesterov's method), coupled with a standard randomized smoothing technique, gives optimal runtime bounds up to an extra multiplicative dependence on d^{1/4} (d is the dimension), which is caused due to the use of the randomized smoothing. They prove the corresponding lower-bound.  In the decentralized setting, the authors use a known primal-dual algorithm which is distributed via a gossip-based algorithm which overall gives optimal runtime (they prove a corresponding tight lower-bound).  Overall, I think the paper presents very solid contributions on problems of definite interest to the optimization community in NIPS. Both upper bounds and nearly matching lower bounds are given, and hence I support its acceptance.  On the technical side, it seems the results are heavily based on previous results, but still I think the current paper has clear contributions.  One thing is, that I think the writing of Section 4 and in particular Subsection 4.2. could be greatly improved and I encourage the authors to work on the presentation. For one, the authors introduce the saddle-point formulation in Eq. 21 which accounts for the constraint that the solutions on all nodes should be equal. However, their algorithm always returns the average of the solutions on all nodes, so why should we enforce this equality constraint at the first-place? a follow-up question, is then why do we need to use the primal-dual method of [14]? I would be very happy to see more technical explanations here to the algorithmic choices/arguments made in the design of Algorithm 2.  Another question regarding the master/slave algorithm: is it trivial to apply a similar technique when the non-smooth functions admit known DETERMINISTIC smoothing schemes (though not necessarily proximal-friendly)?  *** post rebuttal comments *** I have read the response and it answered my questions.