Answers.Body: <p>Yes you could do it but it would be involved.</p>
  <p>Assuming your model is in pytorch you could use <a href="https://pytorch.org/docs/stable/notes/ddp.html" rel="nofollow noreferrer">Distributed Data Parallel</a>.</p>
  <p>All the computers must be nodes on the same network. Network should be fast, so preferably &gt;1Gbps ethernet. A shared storage space would also be beneficial to reduce passing too many data between the nodes. You also need to set up password-less ssh connection between the nodes.</p>
  <p>Then you need to fire up the DDP code for each computer with something like this:</p>
  <pre><code>#!/bin/bash
  
  MASTER_ADDR=&quot;192.168.1.1&quot;
  MASTER_PORT=12355
  WORLD_SIZE=20  # Total number of nodes
  BASE_IP=&quot;192.168.1.&quot; 
  
  # Loop to start nodes
  # Assuming nodes have sequential IPs
  for ( i=1; i&lt;<span class="math-container">$WORLD_SIZE; i++ )
  do
      NODE_IP="$</span>{BASE_IP}<span class="math-container">$i"
      ssh user@$</span>NODE_IP &quot;MASTER_ADDR=<span class="math-container">$MASTER_ADDR MASTER_PORT=$</span>MASTER_PORT WORLD_SIZE=<span class="math-container">$WORLD_SIZE RANK=$</span>i python inference.py&quot; &amp;
  
  wait  # Wait for all nodes to finish
  </code></pre>
  <p>Another option which could be easier to use is &quot;model partioning&quot;. This means that you can load a part of your model that fits on your GPU memory memory, run inference and, write the results to the disk. When one part finishes, offload it and load then next part. This is continued sequentially until the whole process is done.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-06T15:58:46.157Z
Answers.LastActivityDate: 2024-06-06T15:58:46.157Z
Answers.UserDisplayName: sfotiadis
Body: <p>I have no experience with AI whatsoever, and I'm considering creating a relatively simple model just to actually get into AI creation. However, I have a relatively bad GPU (GT 1030) and I'm worried that I won't have enough power to train it. I also may be able to gain access to around 20 computers with Quadro series GPUs, which I don't remember the exact model of, but I know they had 2 GB VRAM. Would there be a way that I could train my model across all of them, despite them being separate computers? Or would my weak GPU be sufficient to train a relatively small model?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-05T07:18:04.067Z
Comments.Text: check [Kubernetes](https://kubernetes.io/).
Comments.UserDisplayName: lpounng
Tags: <neural-networks><training>
Title: How would I train a neural net on multiple different computers?
UserDisplayName: Uranium233
ThreadText: # How would I train a neural net on multiple different computers?
  
  (Asked by: username_0 on 2024-06-05)
  
  I have no experience with AI whatsoever, and I'm considering creating a relatively simple model just to actually get into AI creation. However, I have a relatively bad GPU (GT 1030) and I'm worried that I won't have enough power to train it. I also may be able to gain access to around 20 computers with Quadro series GPUs, which I don't remember the exact model of, but I know they had 2 GB VRAM. Would there be a way that I could train my model across all of them, despite them being separate computers? Or would my weak GPU be sufficient to train a relatively small model?
  
  **Tags:** neural-networks, training
  > **Comments:**
  > **username_1**: check [Kubernetes](https://kubernetes.io/).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-06-06. Score: 1 | Accepted answer
  
  Yes you could do it but it would be involved.
  Assuming your model is in pytorch you could use Distributed Data Parallel.
  All the computers must be nodes on the same network. Network should be fast, so preferably >1Gbps ethernet. A shared storage space would also be beneficial to reduce passing too many data between the nodes. You also need to set up password-less ssh connection between the nodes.
  Then you need to fire up the DDP code for each computer with something like this:
  #!/bin/bash
  
  MASTER_ADDR="192.168.1.1"
  MASTER_PORT=12355
  WORLD_SIZE=20  # Total number of nodes
  BASE_IP="192.168.1." 
  
  # Loop to start nodes
  # Assuming nodes have sequential IPs
  for ( i=1; i<$WORLD_SIZE; i++ )
  do
      NODE_IP="${BASE_IP}$i"
      ssh user@$NODE_IP "MASTER_ADDR=$MASTER_ADDR MASTER_PORT=$MASTER_PORT WORLD_SIZE=$WORLD_SIZE RANK=$i python inference.py" &
  
  wait  # Wait for all nodes to finish
  
  Another option which could be easier to use is "model partioning". This means that you can load a part of your model that fits on your GPU memory memory, run inference and, write the results to the disk. When one part finishes, offload it and load then next part. This is continued sequentially until the whole process is done.
  
  
  ---
  
  