Answers.Body: <p>Check out Figure 6 in this paper: <a href="https://arxiv.org/abs/2006.15704" rel="nofollow noreferrer">PyTorch Distributed: Experiences on Accelerating Data Parallel Training</a></p>
  <p>It breaks down the latency of the forward pass, the backward pass, the communication step, and the optimization step for running both ResNet50 and BERT on a NVIDIA Tesla V100 GPUs.</p>
  <p>From measuring the pixels in the figure, I estimated the times for the forward, backward, and optimization steps as a percentage of their total time combined. (I ignored the communication step shown in the figure because that was only to show how long an unoptimized communication step would take when doing data-parallel training). Here are the estimates I got:</p>
  <ul>
  <li>Forward: 23%</li>
  <li>Backward: 74%</li>
  <li>Optimization: 3%</li>
  </ul>
  <p>So the backward pass takes about 3x as long as the forward pass, and the optimization step is relatively fast.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-28T09:46:25.133Z
Answers.LastActivityDate: 2020-10-01T02:29:38.680Z
Answers.LastEditDate: 2020-10-01T02:29:38.680Z
Answers.UserDisplayName: Elliot Waite
Answers.Body: <h3>Forward pass</h3>
  
  <p>The output of a layer can be calculated given the output of the previous layer. So the GPU can parallelize this computation for every layer and over the minibatch which is done by calculating a big matrix. But it needs to be sequential from layer to layer (earlier layers to higher layers). Regarding the layer type convolutions or especially fully connected layers can result in a big matrix calculation.</p>
  
  <h3>Backward pass</h3>
  
  <p>The gradient of a layer with respect to the layer input (and layer parameters) can only be calculated given the gradient of the layer output (input gradient of a subsequent layer) and input to the layer (output of the previous layer). This again can be parallelized over a layer and minibatch but is sequential from higher layers to earlier layers. Moreover, since the backward pass relies on the outputs of the forward pass all intermediate layer outputs of the forward pass have to be cached for the backward pass which results in a high (GPU) memory usage.</p>
  
  <h3>Forward and backward pass take most of the time</h3>
  
  <p>So, these two steps take a long time for 1 training iteration, and (depending on your network) high GPU memory usage. But you should read and understand the backpropagation algorithm that basically explains everything.</p>
  
  <p>Moreover, to train a network from scratch, in general, takes lots of iterations because especially in the earlier layers training the parameters is based on gradients that are affected by lots of previous layers, which can result in noisy updates, etc., that do not always push the network parameters in the right direction directly. In contrast, e.g. fine-tuning a pre-trained network on some new task can for example already be done with much less training iterations.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-26T19:21:09.223Z
Answers.LastActivityDate: 2020-05-19T19:55:53.987Z
Answers.LastEditDate: 2020-05-19T19:55:53.987Z
Answers.UserDisplayName: Marcel_marcel1991
Answers.Body: <p>There is no such single hard and slow step in training neural networks ,  forward pass involves large number of matrix multiplications so does backward pass , even though there are highly optimized libraries for matrix multiplications neural networks act on very high dimensional (tensor) multiplications in both forward and backward passes which makes it difficult train . however backward pass would be even more slower or even intractable if we won't use backpropagation in case of large neural networks, since computing derivatives is time-taking.</p>
  
  <p>refer training results in for exact number <a href="https://github.com/baidu-research/DeepBench#types-of-operations" rel="nofollow noreferrer">https://github.com/baidu-research/DeepBench#types-of-operations</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-29T15:42:02.713Z
Answers.LastActivityDate: 2018-05-29T15:42:02.713Z
Answers.UserDisplayName: thecomplexitytheorist
Body: <p>Deep networks notoriously take a long time to train. </p>
  
  <p>What is the most time-consuming aspect of training them? Is it the matrix multiplications? Is it the forward pass? Is it some component of the backward pass?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-19T19:52:37.967Z
Comments.Text: See also https://ai.stackexchange.com/q/7865/2444, https://ai.stackexchange.com/q/5728/2444 and https://ai.stackexchange.com/q/13612/2444.
Comments.UserDisplayName: nbro
Tags: <neural-networks><deep-learning><training><deep-neural-networks>
Title: What is the most time-consuming part of training deep networks?
UserDisplayName: echo
ThreadText: # What is the most time-consuming part of training deep networks?
  
  (Asked by: username_0 on 2018-04-01)
  
  Deep networks notoriously take a long time to train.
  
  What is the most time-consuming aspect of training them? Is it the matrix multiplications? Is it the forward pass? Is it some component of the backward pass?
  
  **Tags:** neural-networks, deep-learning, training, deep-neural-networks
  > **Comments:**
  > **username_1**: See also https://ai.stackexchange.com/q/7865/2444, https://ai.stackexchange.com/q/5728/2444 and https://ai.stackexchange.com/q/13612/2444.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2020-09-28. Score: 2
  
  Check out Figure 6 in this paper: [PyTorch Distributed: Experiences on Accelerating Data Parallel Training](https://arxiv.org/abs/2006.15704)
  
  It breaks down the latency of the forward pass, the backward pass, the communication step, and the optimization step for running both ResNet50 and BERT on a NVIDIA Tesla V100 GPUs.
  
  From measuring the pixels in the figure, I estimated the times for the forward, backward, and optimization steps as a percentage of their total time combined. (I ignored the communication step shown in the figure because that was only to show how long an unoptimized communication step would take when doing data-parallel training). Here are the estimates I got:
  
  * Forward: 23%
  * Backward: 74%
  * Optimization: 3%
  
  So the backward pass takes about 3x as long as the forward pass, and the optimization step is relatively fast.
  
  
  ---
  
  ### Answer by username_3 on 2018-09-26. Score: 1
  
  ### Forward pass
  
  The output of a layer can be calculated given the output of the previous layer. So the GPU can parallelize this computation for every layer and over the minibatch which is done by calculating a big matrix. But it needs to be sequential from layer to layer (earlier layers to higher layers). Regarding the layer type convolutions or especially fully connected layers can result in a big matrix calculation.
  
  ### Backward pass
  
  The gradient of a layer with respect to the layer input (and layer parameters) can only be calculated given the gradient of the layer output (input gradient of a subsequent layer) and input to the layer (output of the previous layer). This again can be parallelized over a layer and minibatch but is sequential from higher layers to earlier layers. Moreover, since the backward pass relies on the outputs of the forward pass all intermediate layer outputs of the forward pass have to be cached for the backward pass which results in a high (GPU) memory usage.
  
  ### Forward and backward pass take most of the time
  
  So, these two steps take a long time for 1 training iteration, and (depending on your network) high GPU memory usage. But you should read and understand the backpropagation algorithm that basically explains everything.
  
  Moreover, to train a network from scratch, in general, takes lots of iterations because especially in the earlier layers training the parameters is based on gradients that are affected by lots of previous layers, which can result in noisy updates, etc., that do not always push the network parameters in the right direction directly. In contrast, e.g. fine-tuning a pre-trained network on some new task can for example already be done with much less training iterations.
  
  
  ---
  
  ### Answer by username_4 on 2018-05-29. Score: 0
  
  There is no such single hard and slow step in training neural networks , forward pass involves large number of matrix multiplications so does backward pass , even though there are highly optimized libraries for matrix multiplications neural networks act on very high dimensional (tensor) multiplications in both forward and backward passes which makes it difficult train . however backward pass would be even more slower or even intractable if we won't use backpropagation in case of large neural networks, since computing derivatives is time-taking.
  
  refer training results in for exact number <https://github.com/baidu-research/DeepBench#types-of-operations>
  
  
  ---
  
  