Trading Computation for Communication: Distributed Stochastic Dual Coordinate Ascent
We present and study a distributed optimization algorithm by employing a stochastic dual coordinate ascent method. Stochastic dual coordinate ascent methods enjoy strong theoretical guarantees and often have better performances than stochastic gradient descent methods in optimizing regularized loss minimization problems. It still lacks of efforts in studying them in a distributed framework. We make a progress along the line by presenting a distributed stochastic dual coordinate ascent algorithm in a star network, with an analysis of the tradeoff between computation and communication. We verify our analysis by experiments on real data sets. Moreover, we compare the proposed algorithm with distributed stochastic gradient descent methods and distributed alternating direction methods of multipliers for optimizing SVMs in the same distributed framework, and observe competitive performances.
This paper combines a recent advance in stochastic 
      dual coordinate ascent with the standard approach to parallelize training 
      which is to assign a mini-batch of example to each process and average the 
      resulting gradients. While this combination is novel, its most original 
      contribution is the trade-off bounds between communication and 
      computation. This is likely to be an influential work, introducing a new 
      methodology. The main limitation to its significance are poorly designed 
      experiments. In particular, the authors do not discuss the impact of the 
      input dimension in the bounds, in particular in the case where the data is 
      sparse, which is nearly always the case for big data. 2 major 
      limitations appear in the significance of this trade-off: - In the 
      discussion on page 5, the authors omit the number of dimensions d. If the 
      data is dense, this could be treated as a constant factor as both the 
      communication and computation cost are linear in (I assume the main loop 
      in computation is W.x which is in O(d)). However the situation becomes 
      very different when the data is sparse: the computation cost for a single 
      dot product scales as the number of non-zero features we denote d’. Thus 
      the total computation cost per iteration is O(m*d’). In the worst case, 
      where no features from the m examples overlap, the communication cost will 
      also be O(m*d’). We then lose this m ratio between them that was critical 
      to the analysis on page 5. Note that in the KDD cup data, one feature out 
      of 1 million is non zero. - Experiments only count the number of 
      iterations, not the total running time that includes the communication 
      latency. On top of that, the time per iteration depends on the parameters. 
      So experiments reported on figure 3 that show that increasing m always 
      reduce the number of iteration are very misleading, as each single 
      iteration scales as O(m*d) in computation cost, and O(d) in communication 
      cost. A curve showing the total training time, with values for m and K for 
      which this time is minimized would be much more convincing. In 
      summary, the experiments are so far of little significance, as they only 
      show a reduction of the number of iterations as a function of m and K, 
      which is a trivial observation that does not need theorem 1. Note also 
      that the authors only say they use openMPI, which does say anything about 
      the architecture: - Cluster: how many nodes, how many core per node? 
      - Single multicore? Shared memory would mean no communication costs. 
      The most interesting observation from theorem 1 is the presence of 
      an “effective region” for K and m. But the only thing the experiment show 
      is that decreasing lambda gives more room to choose m and K. A effective 
      upper threshold of the mK product, supported by actual training times, 
      would be a very significant result. Detailed comments: Tens of 
      hundreds of CPU cores: do you mean thousands of cores of tens of clusters 
      with hundreds of cores? If communication costs are involved, the target 
      should be clusters, not multicores with shared memory. Proof of 
      theorem 1: while the proof is correct and trivially obtained from the 
      bound E(epsilon_D^t), the rest of the proof in not needed. In particular, 
      the last sentence is confusing, as there is no T0. “We can complete 
      the proof of Theorem 1 for the smooth loss function by plugging the values 
      of T and T0.” There seems to be several lines that have nothing to do 
      with the proof?? Why do the authors repeat twice DisDCA is 
      parameter free? The choice of lambda is critical. P3, l128: I 
      thought alpha was the associated dual of w, not x, but this may be 
      terminology. Figure 3: varing->varying. Plots in Figure 3 are 
      very hard to read.