Answers.Body: <p>As an example for Caffee: Facebook parallelized the training of a 
  <a href="https://research.fb.com/publications/imagenet1kin1h/" rel="nofollow noreferrer">Res-Net CNN over 256 GPUs</a></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-13T11:50:24.840Z
Answers.LastActivityDate: 2017-06-13T11:50:24.840Z
Answers.UserDisplayName: molig
Answers.Body: <p><em>Disclaimer: this answer refers solely to TensorFlow, as my knowledge of the remaining frameworks is limited.</em></p>
  
  <p>Where did you hear that TensorFlow is near impossible to parallelize on more than 8 GPUs? With a large enough network, any number of GPUs can be used to speed-up training, as shown in the <a href="https://www.tensorflow.org/tutorials/using_gpu" rel="nofollow noreferrer">tutorial</a>. This snippet of code introduces the multiplication of 2 constants in 2 separate GPUs. This allows you to run the graph on both simultaneously.</p>
  
  <pre><code>for d in ['/gpu:2', '/gpu:3']:
      with tf.device(d):
          a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3])
          b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2])
          c.append(tf.matmul(a, b))
  </code></pre>
  
  <p><strong>Even if there were some sort of limit on how many GPUs TensorFlow can use per process, TF can be <a href="https://www.tensorflow.org/deploy/distributed" rel="nofollow noreferrer">distributed</a> across processes and machines.</strong> Asynchronous algorithms (like <a href="https://arxiv.org/abs/1602.01783" rel="nofollow noreferrer">A3C</a>) can be spread across, for example, 16 machines (what I am doing in my laboratory), where each machine uses its resources (in my case, I use CPU, but the change to a GPU would be trivial).</p>
  
  <p>With a complex enough network and a well implemented solution, you can take advantage of any hardware resources you have.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-01T15:23:32.357Z
Answers.Text: Also, in order to be in competition with tensor flow, most other deep learning architectures support this (or have plans to) as well.
Answers.UserDisplayName: Jaden Travnik
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-01T09:06:36.880Z
Answers.LastActivityDate: 2017-06-01T09:06:36.880Z
Answers.UserDisplayName: BlueMoon93
Body: <p>The deep learning algorithms I would to know the limits of are:</p>
  
  <ol>
  <li>CNTK</li>
  <li>Caffe</li>
  <li>TensorFlow</li>
  <li>Torch7</li>
  <li>Theano</li>
  </ol>
  
  <p>For example: I've heard TensorFlow is near impossible to parallelize on 8 GPUs and above. So, in this case, the limit would be 8.</p>
  
Tags: <deep-learning><deep-neural-networks>
Title: How many GPUs can these deep learning algorithms be parallelized across (batch parallelization)?
UserDisplayName: Rushat Rai
ThreadText: # How many GPUs can these deep learning algorithms be parallelized across (batch parallelization)?
  
  (Asked by: username_0 on 2017-05-28)
  
  The deep learning algorithms I would to know the limits of are:
  
  1. CNTK
  2. Caffe
  3. TensorFlow
  4. Torch7
  5. Theano
  
  For example: I've heard TensorFlow is near impossible to parallelize on 8 GPUs and above. So, in this case, the limit would be 8.
  
  **Tags:** deep-learning, deep-neural-networks
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2017-06-01. Score: 2 | Accepted answer
  
  *Disclaimer: this answer refers solely to TensorFlow, as my knowledge of the remaining frameworks is limited.*
  
  Where did you hear that TensorFlow is near impossible to parallelize on more than 8 GPUs? With a large enough network, any number of GPUs can be used to speed-up training, as shown in the [tutorial](https://www.tensorflow.org/tutorials/using_gpu). This snippet of code introduces the multiplication of 2 constants in 2 separate GPUs. This allows you to run the graph on both simultaneously.
  
  ```
  for d in ['/gpu:2', '/gpu:3']:
      with tf.device(d):
          a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3])
          b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2])
          c.append(tf.matmul(a, b))
  
  ```
  
  **Even if there were some sort of limit on how many GPUs TensorFlow can use per process, TF can be [distributed](https://www.tensorflow.org/deploy/distributed) across processes and machines.** Asynchronous algorithms (like [A3C](https://arxiv.org/abs/1602.01783)) can be spread across, for example, 16 machines (what I am doing in my laboratory), where each machine uses its resources (in my case, I use CPU, but the change to a GPU would be trivial).
  
  With a complex enough network and a well implemented solution, you can take advantage of any hardware resources you have.
  
  > **Comments:**
  > **username_3**: Also, in order to be in competition with tensor flow, most other deep learning architectures support this (or have plans to) as well.  
  
  ---
  
  ### Answer by username_1 on 2017-06-13. Score: 2
  
  As an example for Caffee: Facebook parallelized the training of a
  [Res-Net CNN over 256 GPUs](https://research.fb.com/publications/imagenet1kin1h/)
  
  
  ---
  
  