Answers.Body: <p>The whole idea behind those distributed optimization methods is that data should be local in every node/worker. Thus, if you only send the loss value to the central node, this node can't compute the gradients of this loss, and thus can't do any training. However, if you don't want to send gradients, a family of distributed optimization algorithms called consensus-based optimization can be used to only send the local weight of the model to neighbouring nodes, and those nodes use their local gradient and the models from their neighbours to update their local models.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-03T18:47:18.107Z
Answers.LastActivityDate: 2020-01-03T21:17:24.570Z
Answers.LastEditDate: 2020-01-03T21:17:24.570Z
Answers.UserDisplayName: hola
Body: <p>I'm running some distributed trainings in Tensorflow with Horovod. It runs training separately on multiple workers, each of which uses the same weights and does forward pass on unique data. Computed gradients are averaged within the communicator (worker group) before applying them in weight updates. I'm wondering - why not average the loss function across the workers? What's the difference (and the potential benefits) of averaging gradients?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-31T23:39:56.357Z
Comments.Text: It may be because, if you average the losses, you may introduce more numerical errors, which will be propagated to the computation of the gradient, which involves more operations than the computation of the average of the gradients.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-03T17:35:20.910Z
Comments.Text: Following on from the other answer, consider that averaging defeats the purpose if the loss is very volatile. e.g. if one has a low loss reading on one epoch and a high loss reading on another, the average will not capture the volatility in loss readings.
Comments.UserDisplayName: Michael Grogan
Tags: <tensorflow><distributed-computing>
Title: Why do we average gradients and not loss in distributed training?
UserDisplayName: pSoLT
ThreadText: # Why do we average gradients and not loss in distributed training?
  
  (Asked by: username_0 on 2019-12-31)
  
  I'm running some distributed trainings in Tensorflow with Horovod. It runs training separately on multiple workers, each of which uses the same weights and does forward pass on unique data. Computed gradients are averaged within the communicator (worker group) before applying them in weight updates. I'm wondering - why not average the loss function across the workers? What's the difference (and the potential benefits) of averaging gradients?
  
  **Tags:** tensorflow, distributed-computing
  > **Comments:**
  > **username_1**: It may be because, if you average the losses, you may introduce more numerical errors, which will be propagated to the computation of the gradient, which involves more operations than the computation of the average of the gradients.  
  > **username_2**: Following on from the other answer, consider that averaging defeats the purpose if the loss is very volatile. e.g. if one has a low loss reading on one epoch and a high loss reading on another, the average will not capture the volatility in loss readings.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2020-01-03. Score: 2
  
  The whole idea behind those distributed optimization methods is that data should be local in every node/worker. Thus, if you only send the loss value to the central node, this node can't compute the gradients of this loss, and thus can't do any training. However, if you don't want to send gradients, a family of distributed optimization algorithms called consensus-based optimization can be used to only send the local weight of the model to neighbouring nodes, and those nodes use their local gradient and the models from their neighbours to update their local models.
  
  
  ---
  
  