Answers.Body: <p><strong>TL;DR AT BOTTOM</strong></p>
  <p>Depending on your subjective opinion on what a &quot;modern&quot; GPU is, your answer will vary.</p>
  <p>The recent NVIDIA 3000's and 4000's are considered to be &quot;the best&quot; but I cannot find a objective source that gives actual statistics, so this should be taken with a grain of salt.</p>
  <blockquote>
  <p>If I have data with N total inputs, and I sent it through a network of M hidden layers with Nm nodes, and Y outputs. What is the maximum size that a modern GPU can handle running in a reasonable timeframe?</p>
  </blockquote>
  <p>This depends on your AI model. Are you using Tensorflow or the hundreds of other open source projects or are you building from scratch? All AI models try to get the job done, only some do it in a fast timeframe as the main goal.</p>
  <p>In my subjective experience with my &quot;modern&quot; laptop (depending on the definition here) a text generating AI can have up to two million nodes without the laptop being laggy. I have seen <strong>much more</strong> such as ten million nodes on a &quot;normal&quot; (again, subjective!) laptop. Image processing  and more data-full tasks are where millions of nodes are impractical. You would ideally need more than a &quot;modern laptop.&quot; I don't play in this realm and I can't find reliable stats, but for sure you could do more than ten million.</p>
  <p><strong>TL;DR:</strong> Depends on your idea of a &quot;modern device.&quot; I have seen up to ten million nodes on a laptop for text related AI, but no stats exist for GPU's all accross the board.</p>
  <p>Benchmark next time you use the AI model in question to get a good idea of what you can handle.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-12T05:15:48.610Z
Answers.Text: There are some benchmarks and additional information in these links: https://timdettmers.com/2020/09/07/which-gpu-for-deep-learning/#GPU_Deep_Learning_Performance  https://www.run.ai/guides/gpu-deep-learning/best-gpu-for-deep-learning
Answers.UserDisplayName: Rob
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-12T11:46:37.207Z
Answers.Text: @Bapcap, thank you for your answer this is what I was looking for. 
  
  So for this text AI that does two million nodes that you mentioned, is that two million total nodes? Any idea what the dimensions are for the layers?  (how many nodes per layer and how many inputs)
Answers.UserDisplayName: Steven Sagona
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-07T22:57:40.753Z
Answers.Text: @StevenSagona I cannot remember nodes per layer as it differs depending on your model/software. But yes, two million *total* nodes
Answers.UserDisplayName: Bapcap
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-12T01:05:34.523Z
Answers.LastActivityDate: 2022-10-12T01:05:34.523Z
Answers.UserDisplayName: Bapcap
Body: <p>I'm trying to get a general idea of what is a typical size for neural networks that run on a single graphics card.</p>
  <p>What I mean for instance is if I have data with N total inputs, and I sent it through a network of M hidden layers with <span class="math-container">$N_m$</span> nodes, and Y outputs. What is the maximum size that a modern GPU can handle running in a reasonable timeframe?</p>
  <p>I'm just looking for a sort of ballpark understanding of what values are typically used for node number and hidden layer number.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-11T23:35:59.043Z
Comments.Text: "Run" means different things.  "Large" means different things.  "Modern" is as Yoda describes the future: "always in motion".  "Reasonable timeframe" is qualitative.  That said, I have run training on ~50M parameter models in under 48 hours requiring 100k epochs.  If you can convert to lower precision, that can help.  If you use a good framework, that can help.  If your training set isn't too big, that can help.
Comments.UserDisplayName: EngrStudent
Tags: <neural-networks>
Title: How large of a neural network can you run on a modern graphics card?
UserDisplayName: Steven Sagona
ThreadText: # How large of a neural network can you run on a modern graphics card?
  
  (Asked by: username_0 on 2022-10-11)
  
  I'm trying to get a general idea of what is a typical size for neural networks that run on a single graphics card.
  
  What I mean for instance is if I have data with N total inputs, and I sent it through a network of M hidden layers with $N\_m$ nodes, and Y outputs. What is the maximum size that a modern GPU can handle running in a reasonable timeframe?
  
  I'm just looking for a sort of ballpark understanding of what values are typically used for node number and hidden layer number.
  
  **Tags:** neural-networks
  > **Comments:**
  > **username_1**: "Run" means different things.  "Large" means different things.  "Modern" is as Yoda describes the future: "always in motion".  "Reasonable timeframe" is qualitative.  That said, I have run training on ~50M parameter models in under 48 hours requiring 100k epochs.  If you can convert to lower precision, that can help.  If you use a good framework, that can help.  If your training set isn't too big, that can help.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-10-12. Score: 1
  
  **TL;DR AT BOTTOM**
  
  Depending on your subjective opinion on what a "modern" GPU is, your answer will vary.
  
  The recent NVIDIA 3000's and 4000's are considered to be "the best" but I cannot find a objective source that gives actual statistics, so this should be taken with a grain of salt.
  
  > If I have data with N total inputs, and I sent it through a network of M hidden layers with Nm nodes, and Y outputs. What is the maximum size that a modern GPU can handle running in a reasonable timeframe?
  
  This depends on your AI model. Are you using Tensorflow or the hundreds of other open source projects or are you building from scratch? All AI models try to get the job done, only some do it in a fast timeframe as the main goal.
  
  In my subjective experience with my "modern" laptop (depending on the definition here) a text generating AI can have up to two million nodes without the laptop being laggy. I have seen **much more** such as ten million nodes on a "normal" (again, subjective!) laptop. Image processing and more data-full tasks are where millions of nodes are impractical. You would ideally need more than a "modern laptop." I don't play in this realm and I can't find reliable stats, but for sure you could do more than ten million.
  
  **TL;DR:** Depends on your idea of a "modern device." I have seen up to ten million nodes on a laptop for text related AI, but no stats exist for GPU's all accross the board.
  
  Benchmark next time you use the AI model in question to get a good idea of what you can handle.
  
  > **Comments:**
  > **username_3**: There are some benchmarks and additional information in these links: https://timdettmers.com/2020/09/07/which-gpu-for-deep-learning/#GPU_Deep_Learning_Performance  https://www.run.ai/guides/gpu-deep-learning/best-gpu-for-deep-learning  
  > **username_0**: @Bapcap, thank you for your answer this is what I was looking for.   So for this text AI that does two million nodes that you mentioned, is that two million total nodes? Any idea what the dimensions are for the layers?  (how many nodes per layer and how many inputs)  
  > **username_2**: @StevenSagona I cannot remember nodes per layer as it differs depending on your model/software. But yes, two million *total* nodes  
  
  ---
  
  