Distributed training of deep learning is widely conducted with large neural networks and large datasets. Besides asynchronous stochastic gradient descent (SGD), synchronous SGD is a reasonable alternative with better convergence guarantees. However, synchronous SGD suffers from stragglers. To make things worse, although there are some strategies dealing with slow workers, the issue of slow servers is commonly ignored. In this paper, we propose a new parameter server (PS) framework dealing with not only slow workers, but also slow servers by weakening the synchronization criterion. The empirical results show good performance when there are stragglers.
This paper introduces a parameter server architecture to improve distributed training of CNNs in the presence of stragglers. Specifically, the paper proposes partial pulling where a worker only waits for first b blocks rather than all the blocks of the parameters. This technique is combined with existing methods such as partial pushing (Pan et. al. 2017) for a partial synchronous SGD method. The method is evaluated with Resnet -50 using synthetic delays.
Comments for the author:
The paper is well-written and easy to follow. The problem of synchronization costs being addressed is important but it is unclear how much of this is arising due to large blocks.
1) The partial pushing method (Pan et. al. 2017, section 3.1) shows a clear evidence for the problem using a real workload with a large number of workers. Unfortunately, in your Figure 2, this is not as obvious and not real since it is using simulated delays. More specifically, it is not clear how the workers behave in a real environment and whether you get a clear benefit from using a partial number of blocks as opposed to sending all of them. 
2) Did you modify your code to support block-wise sending of gradients (some description of how the framework was modified will be helpful)? The idea is to send partial parameter blocks and when 'b' blocks are received, compute the gradients. I feel that, with such a design, you may actually end up hurting the performance by sending a large number of small packets in the no failure case. For real, large data centers, this may cause a packet storm and subsequent throughput collapse (e.g. the incast problem). You need to show the evidence that you do not hurt the failure-free case for a large number of workers.
3) The evaluation is on fairly small workloads (CIFAR-10). Again, evaluating over Imagenet and demonstrating a clear speedup over existing sync methods will be helpful. Furthermore, a clear description of your “pull” configuration (such as in Figure 1) i.e. how many actual bytes or blocks are sent and what is the threshold will be helpful (beyond a vague 90%).
4) Another concern with partial synchronization methods that I have is that how do you pick these configurations (pull 0.75 etc). These appear to be dataset specific and finding the optimal configuration here requires significant experimentation that takes significantly more time than just running the baseline.
Overall, I feel there is not enough evidence for the problem specifically generating large blocks of gradients and this needs to be clearly shown. To propose a solution for stragglers, evaluation should be done in a datacenter environment with the presence of stragglers (and not small workloads with synthetic delays). Furthermore, the proposed technique despite the simplicity appears as a rather incremental contribution.