Body: <p>I am trying to figure out how multiprocessing works in neural networks.</p>
  <p>In the example I've seen, the database is split into <span class="math-container">$x$</span> parts (depending on how many workers you have) and each worker is responsible to train the network using a different part of the database.</p>
  <p>I am confused regarding the optimization part:</p>
  <p>Let's say worker 1 finished first to calculate the gradient, now it will update the network accordingly.</p>
  <p>Then worker 2 finished the calculation and it will also attempt to update the weights. However, the gradient it calculated was for the network before it was updated by the first worker. Now, the second worker will attempt to update the network with a bad gradient.</p>
  <p>Did I miss something?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-19T13:33:35.073Z
Comments.Text: depending on your ML framework (e.g, TF, pytorch), beside the docs they have pretty nice explanation on several of their ways of distributed training. ANW, check out the Hogwild! training https://www.youtube.com/watch?v=l5JqUvTdZts
Comments.UserDisplayName: Sanyou
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-19T15:00:00.177Z
Comments.Text: multiprocessing or gpus allows you to process several batches at the same time, but indeed gradients and optimizer can be updated only after all relevant batches have been processed. So usually one process or one gpu is reserved for this specific task. Pytorch for example collect all gradients always in cuda0 (the first, or the only, gpu available).
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-22T09:31:04.530Z
Comments.Text: If you think about it this is isn't too different from how non-multiprocessing training works - we calculate the gradient for a whole lot of data items, without updating the network, then we update the network with all the gradients at once!
Comments.UserDisplayName: Criticize SE actions means ban
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-22T13:04:24.920Z
Comments.Text: @Sanyou This is an important question, although it's quite general, given that there may be different ways to do this. If you can provide a formal answer, I would suggest that you do so, because it may be useful to other people in the future too.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-22T13:05:44.450Z
Comments.Text: @EdoardoGuerriero I wanted to write the comment above by also mentioning you, but I couldn't do that. It seems that you have some useful information that at least partially answers the question. Maybe you can elaborate on what you wrote above to also write a formal answer below.
Comments.UserDisplayName: nbro
Tags: <neural-networks><gradient-descent><gpu>
Title: How to train neural networks with multiprocessing?
UserDisplayName: Yedidya kfir
ThreadText: # How to train neural networks with multiprocessing?
  
  (Asked by: username_0 on 2021-10-19)
  
  I am trying to figure out how multiprocessing works in neural networks.
  
  In the example I've seen, the database is split into $x$ parts (depending on how many workers you have) and each worker is responsible to train the network using a different part of the database.
  
  I am confused regarding the optimization part:
  
  Let's say worker 1 finished first to calculate the gradient, now it will update the network accordingly.
  
  Then worker 2 finished the calculation and it will also attempt to update the weights. However, the gradient it calculated was for the network before it was updated by the first worker. Now, the second worker will attempt to update the network with a bad gradient.
  
  Did I miss something?
  
  **Tags:** neural-networks, gradient-descent, gpu
  
  > **Comments:**
  > **username_1**: depending on your ML framework (e.g, TF, pytorch), beside the docs they have pretty nice explanation on several of their ways of distributed training. ANW, check out the Hogwild! training https://www.youtube.com/watch?v=l5JqUvTdZts  
  > **username_2**: multiprocessing or gpus allows you to process several batches at the same time, but indeed gradients and optimizer can be updated only after all relevant batches have been processed. So usually one process or one gpu is reserved for this specific task. Pytorch for example collect all gradients always in cuda0 (the first, or the only, gpu available).  
  > **username_3**: If you think about it this is isn't too different from how non-multiprocessing training works - we calculate the gradient for a whole lot of data items, without updating the network, then we update the network with all the gradients at once!  
  > **username_4**: @username_1 This is an important question, although it's quite general, given that there may be different ways to do this. If you can provide a formal answer, I would suggest that you do so, because it may be useful to other people in the future too.  
  > **username_4**: @username_2 I wanted to write the comment above by also mentioning you, but I couldn't do that. It seems that you have some useful information that at least partially answers the question. Maybe you can elaborate on what you wrote above to also write a formal answer below.  
  
  ---
  
  ## 0 answers
  
  