title: Asynchronous Local-SGD Training for Language Modeling
summary: Local stochastic gradient descent (Local-SGD), also referred to as federated
  averaging, is an approach to distributed optimization where each device
  performs more than one SGD update per communication. This work presents an
  empirical study of {\it asynchronous} Local-SGD for training language models;
  that is, each worker updates the global parameters as soon as it has finished
  its SGD steps. We conduct a comprehensive investigation by examining how worker
  hardware heterogeneity, model size, number of workers, and optimizer could
  impact the learning performance. We find that with naive implementations,
  asynchronous Local-SGD takes more iterations to converge than its synchronous
  counterpart despite updating the (global) model parameters more frequently. We
  identify momentum acceleration on the global parameters when worker gradients
  are stale as a key challenge. We propose a novel method that utilizes a delayed
  Nesterov momentum update and adjusts the workers' local training steps based on
  their computation speed. This approach, evaluated with models up to 150M
  parameters on the C4 dataset, matches the performance of synchronous Local-SGD
  in terms of perplexity per update step, and significantly surpasses it in terms
  of wall clock time.
paper_authors._id: 65a8a864fee7ba6063c7e431
paper_authors.name: Bo Liu
paper_authors._id: 65a8a864fee7ba6063c7e432
paper_authors.name: Rachita Chhaparia
paper_authors._id: 65a8a864fee7ba6063c7e433
paper_authors.name: Arthur Douillard
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-18T10:06:56.510Z
paper_authors._id: 622792366303bf1dc304f49f
paper_authors.avatarUrl: /avatars/975c1cc3eb2f97cf8e848162056d5bea.svg
paper_authors.fullname: Arthur Douillard
paper_authors.type: user
paper_authors.user: ArthurDouillard
paper_authors._id: 65a8a864fee7ba6063c7e434
paper_authors.name: Satyen Kale
paper_authors._id: 65a8a864fee7ba6063c7e435
paper_authors.name: Andrei A. Rusu
paper_authors._id: 65a8a864fee7ba6063c7e436
paper_authors.name: Jiajun Shen
paper_authors._id: 65a8a864fee7ba6063c7e437
paper_authors.name: Arthur Szlam
paper_authors._id: 65a8a864fee7ba6063c7e438
paper_authors.name: Marc'Aurelio Ranzato
paper_title: Asynchronous Local-SGD Training for Language Modeling
paper_summary: Local stochastic gradient descent (Local-SGD), also referred to as federated
  averaging, is an approach to distributed optimization where each device
  performs more than one SGD update per communication. This work presents an
  empirical study of {\it asynchronous} Local-SGD for training language models;
  that is, each worker updates the global parameters as soon as it has finished
  its SGD steps. We conduct a comprehensive investigation by examining how worker
  hardware heterogeneity, model size, number of workers, and optimizer could
  impact the learning performance. We find that with naive implementations,
  asynchronous Local-SGD takes more iterations to converge than its synchronous
  counterpart despite updating the (global) model parameters more frequently. We
  identify momentum acceleration on the global parameters when worker gradients
  are stale as a key challenge. We propose a novel method that utilizes a delayed
  Nesterov momentum update and adjusts the workers' local training steps based on
  their computation speed. This approach, evaluated with models up to 150M
  parameters on the C4 dataset, matches the performance of synchronous Local-SGD
  in terms of perplexity per update step, and significantly surpasses it in terms
  of wall clock time.
paper_ai_keywords: Local stochastic gradient descent (Local-SGD) federated averaging asynchronousLocal-SGD worker hardware heterogeneity model size number of workers optimistic momentum acceleration Nesterov momentum update C4 dataset perplexity