Distributed computing can significantly reduce the training time of neural networks. Despite its potential, however, distributed training has not been widely adopted: scaling the training process is difficult, and existing SGD methods require substantial tuning of hyperparameters and learning schedules to achieve sufficient accuracy when increasing the number of workers. In practice, such tuning can be prohibitively expensive given the huge number of potential hyperparameter configurations and the effort required to test each one. We propose DANA, a novel approach that scales out-of-the-box to large clusters using the same hyperparameters and learning schedule optimized for training on a single worker, while maintaining similar final accuracy without additional overhead. DANA estimates the future value of model parameters by adapting Nesterov Accelerated Gradient to a distributed setting, and so mitigates the effect of gradient staleness, one of the main difficulties in scaling SGD to more workers. Evaluation on three state-of-the-art network architectures and three datasets shows that DANA scales as well as or better than existing work without having to tune any hyperparameters or tweak the learning schedule. For example, DANA achieves 75.73% accuracy on ImageNet when training ResNet-50 with 16 workers, similar to the non-distributed baseline.
# overview
In this work, Nesterov Accelerated gradient based updates are applied in a distributed fashion to scale SGD based training to multiple nodes without the introduction of further hyperparameters or having to adapt the learning rate schedule from that of single node training on the same data.
Evaluation is carried out on image classification workloads using ResNet model variants across CIFAR10,100, and ImageNet datasets, utilizing from 8-32 nodes. In contrasting test error relative to single node performance, the authors find their method degrades less than other synchronous and asynchronous SGD based approaches as node count increases.
Overall, this work is presented in a fairly clear and logical manner, and the writing is easy to follow.  However the approach described appears to be contingent on very specific worker communication patterns and timing which seem unrealistic for real-world settings (namely that each worker sends exactly one update per N sized block received).  Extrapolating from the curvature of the results shown it doesn't appear that DANA would continue to outperform other methods like ASGD once the worker count scales beyond the 32 node limit evaluated.
# pros
* no additional hyperparameter tuning required
* should be easy to drop into existing asynchronous SGD implementations, just need to modify the worker side.
* does appear to scale slightly better from an accuracy perspective in 16-32 node counts
# cons
* Biggest criticism is the assumption of block random or round-robin worker update scheduling. Presuming each worker will update master exactly once to determine future parameter position is far from realistic on real hardware (varying capacity, performance, system loads, dealing with stragglers) and should probably be considered a synchronous not asynchronous update.
* only evaluated on image classification tasks on cifar10, cifar100, imagenet on resnet-20 and resnet-50. Would have been better to evaluate on a more varied set of tasks/models/datasets
# other comments
* Figure 2 baseline performance reported is a bit misleading/confusing since it was only evaluated on a single worker. Would suggest restricting to a single point rather than some extrapolated line that seems to indicate being run on multiple-workers.
* Figure 3 should should also show multi-node speedups for the other methods compared for completeness. 
* Section 5.2 should report on percentage scaling efficiency rather than using speedup as it doesn't normalize for worker count.  For instance 16x could be interpreted as good or poor if it was achieved using 16 vs 160 nodes.
* Section 5.2 there's a small typo: GPUs -> GPU
* Consider https://arxiv.org/abs/1705.07176 in related work?