Answers.Body: <p>GPUs are able to execute a huge amount of similar and simple instructions (floating point operations like addition and multiplication) in parallel. In contrast to a CPU which is able to execute a few complex tasks sequentially very quick. Therefore GPUs are very good at doing vector &amp; matrix operations.</p>
  
  <p>If you look at the operations performed inside a single basic NN layer you will see that most of the operations are matrix-vector multiplications:</p>
  
  <p><span class="math-container">$$x_{i+1} = \sigma(W_ix_i + b_i)$$</span></p>
  
  <p>where <span class="math-container">$x_i$</span> is the input vector, <span class="math-container">$W_i$</span> the matrix of weights, <span class="math-container">$b_i$</span> the bias vector in the <span class="math-container">$i^{th}$</span> layer, <span class="math-container">$x_{i+1}$</span> the output vector and <span class="math-container">$\sigma(\cdot)$</span> the elemtwise non-linear activation. The computational complexity here is governed by the matrix-vector multiplication. If you look at the architecture of a <a href="https://colah.github.io/posts/2015-08-Understanding-LSTMs/" rel="nofollow noreferrer">LSTM cell</a> you will notice that inside of it are multiple such operations.</p>
  
  <p>Being able to execute the matrix-vector operations quickly and efficiently in parallel will reduce execution time, this is where GPUs excel CPUs.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-16T14:13:39.370Z
Answers.Text: This is quite insightful. You have pointed that CPUs are slow because they are capable of doing sequential tasks at best. But, to my understanding, I believe CPUs are slow because of less number of cores. A typical CPU constitutes 4 cores, whereas Tesla K80 (easily available general purpose GPU) has 4992 CUDA Cores. Hence, if we employ a CPU with comparable cores to K80, we might get same performance. What are your thoughts in this?
Answers.UserDisplayName: Anubhav Sachan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-17T03:00:46.660Z
Answers.Text: @AnubhavSachan You wouldn't get the same performance, you'd get much better performance. CPUs are designed to have very fast single thread speed, but the cost/thread is much higher. So for $1000 you could buy 4 CPU cores, but could buy 5000 GPU cores. It all depends on the situation which is better. But if you had the money, and wanted the absolute fastest performance, you'd saturate the calculations with CPU cores (by saturate I mean more cores wont help because there's only so much you can compute).
Answers.UserDisplayName: Recessive
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-17T08:28:38.883Z
Answers.Text: @Recessive Very nice addition to my answer! Thanks a lot! The cost/thread aspect is of course one of the main pain points here. I forgot to mention it.
Answers.UserDisplayName: Tinu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-17T16:17:00.657Z
Answers.Text: Great answers! Thanks for your inputs. This question turned out to be a good discussion.
Answers.UserDisplayName: Anubhav Sachan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-03T18:20:30.950Z
Answers.Text: someone has a reference of a book or paper that explain the Tinu's answer?
Answers.UserDisplayName: Xtalker
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-16T13:46:44.857Z
Answers.LastActivityDate: 2020-06-16T14:04:29.503Z
Answers.LastEditDate: 2020-06-16T14:04:29.503Z
Answers.UserDisplayName: Tinu
Body: <p>I would love to know in detail, how exactly GPUs help, in technical terms, in training the deep learning models.</p>
  
  <p>To my understanding, GPUs help in performing independent tasks simultaneously to improve the speed. For example, in calculation of the output through CNN, all the additions are done simultaneously, and hence improves the speed.</p>
  
  <p>But, what exactly happens in a basic neural network or in a LSTM type complex models in regard with GPU.</p>
  
Tags: <neural-networks><deep-learning><convolutional-neural-networks><long-short-term-memory><gpu>
Title: How do GPUs faciliate the training of a Deep Learning Architecture?
UserDisplayName: Anubhav Sachan
ThreadText: # How do GPUs faciliate the training of a Deep Learning Architecture?
  
  (Asked by: username_0 on 2020-06-16)
  
  I would love to know in detail, how exactly GPUs help, in technical terms, in training the deep learning models.
  
  To my understanding, GPUs help in performing independent tasks simultaneously to improve the speed. For example, in calculation of the output through CNN, all the additions are done simultaneously, and hence improves the speed.
  
  But, what exactly happens in a basic neural network or in a LSTM type complex models in regard with GPU.
  
  **Tags:** neural-networks, deep-learning, convolutional-neural-networks, long-short-term-memory, gpu
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-16. Score: 3 | Accepted answer
  
  GPUs are able to execute a huge amount of similar and simple instructions (floating point operations like addition and multiplication) in parallel. In contrast to a CPU which is able to execute a few complex tasks sequentially very quick. Therefore GPUs are very good at doing vector & matrix operations.
  
  If you look at the operations performed inside a single basic NN layer you will see that most of the operations are matrix-vector multiplications:
  
  $$x\_{i+1} = \sigma(W\_ix\_i + b\_i)$$
  
  where $x\_i$ is the input vector, $W\_i$ the matrix of weights, $b\_i$ the bias vector in the $i^{th}$ layer, $x\_{i+1}$ the output vector and $\sigma(\cdot)$ the elemtwise non-linear activation. The computational complexity here is governed by the matrix-vector multiplication. If you look at the architecture of a [LSTM cell](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) you will notice that inside of it are multiple such operations.
  
  Being able to execute the matrix-vector operations quickly and efficiently in parallel will reduce execution time, this is where GPUs excel CPUs.
  
  > **Comments:**
  > **username_0**: This is quite insightful. You have pointed that CPUs are slow because they are capable of doing sequential tasks at best. But, to my understanding, I believe CPUs are slow because of less number of cores. A typical CPU constitutes 4 cores, whereas Tesla K80 (easily available general purpose GPU) has 4992 CUDA Cores. Hence, if we employ a CPU with comparable cores to K80, we might get same performance. What are your thoughts in this?  
  > **username_2**: @AnubhavSachan You wouldn't get the same performance, you'd get much better performance. CPUs are designed to have very fast single thread speed, but the cost/thread is much higher. So for $1000 you could buy 4 CPU cores, but could buy 5000 GPU cores. It all depends on the situation which is better. But if you had the money, and wanted the absolute fastest performance, you'd saturate the calculations with CPU cores (by saturate I mean more cores wont help because there's only so much you can compute).  
  > **username_1**: @Recessive Very nice addition to my answer! Thanks a lot! The cost/thread aspect is of course one of the main pain points here. I forgot to mention it.  
  > **username_0**: Great answers! Thanks for your inputs. This question turned out to be a good discussion.  
  > **username_3**: someone has a reference of a book or paper that explain the Tinu's answer?  
  
  ---
  
  