Definition: In this task, you are given a part of an article. Your task is to generate headline (title) for this text. Preferred headlines are under fifteen words.
Input: <lb>We study the problem of stochastic optimization for deep learning in the paral-<lb>lel computing environment under communication constraints. A new algorithm<lb>is proposed in this setting where the communication and coordination of work<lb>among concurrent processes (local workers), is based on an elastic force which<lb>links the parameters they compute with a center variable stored by the parameter<lb>server (master). The algorithm enables the local workers to perform more explo-<lb>ration, i.e. the algorithm allows the local variables to fluctuate further from the<lb>center variable by reducing the amount of communication between local workers<lb>and the master. We empirically demonstrate that in the deep learning setting, due<lb>to the existence of many local optima, allowing more exploration can lead to the<lb>improved performance. We propose synchronous and asynchronous variants of<lb>the new algorithm. We provide the stability analysis of the asynchronous vari-<lb>ant in the round-robin scheme and compare it with the more common parallelized<lb>method ADMM. We show that the stability of EASGD is guaranteed when a simple<lb>stability condition is satisfied, which is not the case for ADMM. We additionally<lb>propose the momentum-based version of our algorithm that can be applied in both<lb>synchronous and asynchronous settings. Asynchronous variant of the algorithm<lb>is applied to train convolutional neural networks for image classification on the<lb>CIFAR and ImageNet datasets. Experiments demonstrate that the new algorithm<lb>accelerates the training of deep architectures compared to DOWNPOUR and other<lb>common baseline approaches and furthermore is very communication efficient.

Deep learning with Elastic Averaging SGD