Answers.Body: <p>As an example for Caffee: Facebook parallelized the training of a 
  <a href="https://research.fb.com/publications/imagenet1kin1h/" rel="nofollow noreferrer">Res-Net CNN over 256 GPUs</a></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-13T11:50:24.840Z
Answers.LastActivityDate: 2017-06-13T11:50:24.840Z
Answers.UserDisplayName: molig
Answers.Body: <p><em>Disclaimer: this answer refers solely to TensorFlow, as my knowledge of the remaining frameworks is limited.</em></p>
  
  <p>Where did you hear that TensorFlow is near impossible to parallelize on more than 8 GPUs? With a large enough network, any number of GPUs can be used to speed-up training, as shown in the <a href="https://www.tensorflow.org/tutorials/using_gpu" rel="nofollow noreferrer">tutorial</a>. This snippet of code introduces the multiplication of 2 constants in 2 separate GPUs. This allows you to run the graph on both simultaneously.</p>
  
  <pre><code>for d in ['/gpu:2', '/gpu:3']:
      with tf.device(d):
          a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3])
          b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2])
          c.append(tf.matmul(a, b))
  </code></pre>
  
  <p><strong>Even if there were some sort of limit on how many GPUs TensorFlow can use per process, TF can be <a href="https://www.tensorflow.org/deploy/distributed" rel="nofollow noreferrer">distributed</a> across processes and machines.</strong> Asynchronous algorithms (like <a href="https://arxiv.org/abs/1602.01783" rel="nofollow noreferrer">A3C</a>) can be spread across, for example, 16 machines (what I am doing in my laboratory), where each machine uses its resources (in my case, I use CPU, but the change to a GPU would be trivial).</p>
  
  <p>With a complex enough network and a well implemented solution, you can take advantage of any hardware resources you have.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-01T15:23:32.357Z
Answers.Text: Also, in order to be in competition with tensor flow, most other deep learning architectures support this (or have plans to) as well.
Answers.UserDisplayName: Jaden Travnik
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-01T09:06:36.880Z
Answers.LastActivityDate: 2017-06-01T09:06:36.880Z
Answers.UserDisplayName: BlueMoon93
Body: <p>The deep learning algorithms I would to know the limits of are:</p>
  
  <ol>
  <li>CNTK</li>
  <li>Caffe</li>
  <li>TensorFlow</li>
  <li>Torch7</li>
  <li>Theano</li>
  </ol>
  
  <p>For example: I've heard TensorFlow is near impossible to parallelize on 8 GPUs and above. So, in this case, the limit would be 8.</p>
  
Tags: <deep-learning><deep-neural-networks>
Title: How many GPUs can these deep learning algorithms be parallelized across (batch parallelization)?
UserDisplayName: Rushat Rai
ThreadText: # How many GPUs can these deep learning algorithms be parallelized across (batch parallelization)?
  
  (Asked by: username_0 on 2017-05-28)
  
  The deep learning algorithms I would to know the limits of are:
  
  1. CNTK
  2. Caffe
  3. TensorFlow
  4. Torch7
  5. Theano
  
  For example: I've heard TensorFlow is near impossible to parallelize on 8 GPUs and above. So, in this case, the limit would be 8.
  
  **Tags:** deep-learning, deep-neural-networks
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2017-06-01. Score: 2 | Accepted answer
  
  Disclaimer: this answer refers solely to TensorFlow, as my knowledge of the remaining frameworks is limited.
  Where did you hear that TensorFlow is near impossible to parallelize on more than 8 GPUs? With a large enough network, any number of GPUs can be used to speed-up training, as shown in the tutorial. This snippet of code introduces the multiplication of 2 constants in 2 separate GPUs. This allows you to run the graph on both simultaneously.
  for d in ['/gpu:2', '/gpu:3']:
      with tf.device(d):
          a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3])
          b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2])
          c.append(tf.matmul(a, b))
  
  Even if there were some sort of limit on how many GPUs TensorFlow can use per process, TF can be distributed across processes and machines. Asynchronous algorithms (like A3C) can be spread across, for example, 16 machines (what I am doing in my laboratory), where each machine uses its resources (in my case, I use CPU, but the change to a GPU would be trivial).
  With a complex enough network and a well implemented solution, you can take advantage of any hardware resources you have.
  
  > **Comments:**
  > **username_3**: Also, in order to be in competition with tensor flow, most other deep learning architectures support this (or have plans to) as well.  
  
  ---
  
  ### Answer by username_1 on 2017-06-13. Score: 2
  
  As an example for Caffee: Facebook parallelized the training of a
  [Res-Net CNN over 256 GPUs](https://research.fb.com/publications/imagenet1kin1h/)
  
  
  ---
  
  