Answers.Body: <p>As a caveat, I’d suggest that unless you’re pushing up against fundamental technological limits, computation speed and resources should be secondary to design rationale when developing a neural network architecture.</p>
  
  <p>That said, earlier this year I finished my MS thesis that involved bioinformatics analytics pipelines with whole genome sequencing data - that project took over 100,000 hours of compute time to develop according to our clusters job manager. When your on a deadline, resources can be a real constraint and speed can be critical. </p>
  
  <p>So, to answer your questions as I understand them:</p>
  
  <p><strong>Would I have been better off to use the money to buy time in the cloud?</strong></p>
  
  <p>Probably. The few hundred dollars you spent on the 1060 would take you far training your model(s) in the cloud. Further, as far as I can tell, you don’t require the GPU to be cranking 100% of the time (you would if you were, say, mining crypto currencies). Finally, with cloud instances you could scale, training multiple models at once, which can speed up the exploration and validation of any architecture you settle on. </p>
  
  <p><strong>Is there a way to gauge the compute time of a neural network on a given GPU</strong></p>
  
  <p>Well, <a href="https://en.m.wikipedia.org/wiki/Big_O_notation" rel="nofollow noreferrer">Big O</a> is one estimator, but it sounds like you want a more precise method. I’m sure they exist, but I’d counter that you can make your estimation with simple back of the envelope calculations that account for threads, memory, code iterations, etc. Do you really want to dig into the GPU processing pipeline on the 1060? You may be able to come up with a very good estimate by understanding everything happening between your code and the metal, but ultimately it’s probably not worth the time and effort; it will likely confirm that Big O notation (the simple model, if you will) captures most of the variation in compute time. One thing you can do if you notice bottlenecks is performance profiling. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-13T19:40:21.683Z
Answers.LastActivityDate: 2017-12-13T19:45:48.340Z
Answers.LastEditDate: 2017-12-13T19:45:48.340Z
Answers.UserDisplayName: Greenstick
Answers.Body: <p>It depends on what you need. You can train any size of network on any resource. The problem is the time of training. If you want to train Inception on an average CPU it will take months to converge. So, it all depends on how long you can wait to see your results based on your network. As in neural nets we do not have only one operation but many (like concatenating, max pooling, padding etc.), it is impossible to make an estimation as you are searching for. Just start training some infamous networks and measure the time. Then, you can interpolate how long it will take to train networks that you are searching for. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-13T21:13:58.813Z
Answers.Text: Are you aware of a resource that collects information for different NN jobs? So a list of: NN type, unit count, hyperparameters, dataset size, Hardware used, time taken? That would help to develop a rule of thumb intuition
Answers.UserDisplayName: pascalwhoop
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-13T03:38:52.177Z
Answers.LastActivityDate: 2017-12-13T03:38:52.177Z
Answers.UserDisplayName: Deniz Beker
Answers.Body: <p>Usually the problem is to fit the model into video RAM.
  If it does not, you cannot train your model at all without big efforts (like training parts of the model separately).
  If it does, time is your only problem. But the difference in training time between consumer GPUs like the Nvidia 1080 and much more expensive GPU accelerators like the Nvidia K80 are not very large. Actually the best consumer cards are faster than GPU accelerators, but lack other properties like VRAM. Random comparisons and benchmarks: <a href="http://timdettmers.com/2017/04/09/which-gpu-for-deep-learning/" rel="nofollow noreferrer">Which GPU(s) to Get for Deep Learning: My Experience and Advice for Using GPUs in Deep Learning</a>, <a href="https://medium.com/@alexbaldo/a-comparison-between-nvidias-geforce-gtx-1080-and-tesla-p100-for-deep-learning-81a918d5b2c7" rel="nofollow noreferrer">A Comparison between NVIDIA’s GeForce GTX 1080 and Tesla P100 for Deep Learning</a>.</p>
  <p>To calculate if your models fits into VRAM, you just approximate how much data and which hyperparameters you have (inputs, outputs, weights, layers, batch size, which datatype and so on).</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-16T14:28:00.333Z
Answers.Text: Good answer, but what kind of a model does not fit into a 6/8GB Memory? Let's say each batch of training data + the model. I have no idea how to calculate the size of the model but I cannot imagine a few thousands neurons and their weights can fill up Gigabytes of Memory. The Pretrained image recognition model from Google is a few hundred Megs https://www.tensorflow.org/tutorials/image_recognition and that can recognize more than most others out there. Apple handles it on-chip in the iPhone so the model itself must be much smaller than the T-data. And that doesn't need to be all in RAM right?
Answers.UserDisplayName: pascalwhoop
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-18T03:55:05.450Z
Answers.Text: I am often working with NLP models, mostly multi-layer RNNs, and if the vocabulary size is large, even 8 GB might not be enough.
  
  Remember that training usually needs more space than prediction (gradients take up space also).
Answers.UserDisplayName: C. Yduqoli
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-12-15T09:30:20.337Z
Answers.LastActivityDate: 2021-04-23T01:39:47.150Z
Answers.LastEditDate: 2021-04-23T01:39:47.150Z
Answers.UserDisplayName: C. Yduqoli
Body: <p>Is it possible to give a <strong>rule of thumb estimate about the size of neural networks that are trainable on common consumer-grade GPUs</strong>?</p>
  <p>For example, the <a href="https://arxiv.org/abs/1707.02286" rel="nofollow noreferrer">Emergence of Locomotion (Reinforcement)</a> paper trains a network using tanh activation of the neurons. They have a 3 layer NN with 300,200,100 units for the <em>Planar Walker</em>. But they don’t report the hardware and time.</p>
  <p>But could a rule of thumb be developed?</p>
  <p>Also, just based on current empirical results. So, for example, <span class="math-container">$X$</span> units using sigmoid activation can run <span class="math-container">$Y$</span> learning iterations per hour on a 1060.</p>
  <p>Or using activation function <span class="math-container">$a$</span> instead of <span class="math-container">$b$</span> causes a <span class="math-container">$n$</span> times decrease in performance.</p>
  <p>If a student/researcher/curious mind is going to buy a GPU for playing around with these networks, how do you decide what you get? A 1060 is apparently the entry-level budget option, but how can you evaluate if it is not smarter to just get a crappy netbook instead of building a high-power desktop and spend the saved $ on on-demand cloud infrastructure.</p>
  <p>Motivation for the question: I just purchased a 1060 and (clever, to ask the question afterwards huh) wonder if I should have just kept the $ and made a Google Cloud account. And if I can run my master thesis simulation on the GPU.</p>
  
Tags: <neural-networks><training><performance><hardware-evaluation>
Title: What size of neural networks can be trained on current consumer grade GPUs? (1060,1070,1080)
UserDisplayName: pascalwhoop
ThreadText: # What size of neural networks can be trained on current consumer grade GPUs? (1060,1070,1080)
  
  (Asked by: username_0 on 2017-12-09)
  
  Is it possible to give a **rule of thumb estimate about the size of neural networks that are trainable on common consumer-grade GPUs**?
  
  For example, the [Emergence of Locomotion (Reinforcement)](https://arxiv.org/abs/1707.02286) paper trains a network using tanh activation of the neurons. They have a 3 layer NN with 300,200,100 units for the *Planar Walker*. But they don’t report the hardware and time.
  
  But could a rule of thumb be developed?
  
  Also, just based on current empirical results. So, for example, $X$ units using sigmoid activation can run $Y$ learning iterations per hour on a 1060.
  
  Or using activation function $a$ instead of $b$ causes a $n$ times decrease in performance.
  
  If a student/researcher/curious mind is going to buy a GPU for playing around with these networks, how do you decide what you get? A 1060 is apparently the entry-level budget option, but how can you evaluate if it is not smarter to just get a crappy netbook instead of building a high-power desktop and spend the saved $ on on-demand cloud infrastructure.
  
  Motivation for the question: I just purchased a 1060 and (clever, to ask the question afterwards huh) wonder if I should have just kept the $ and made a Google Cloud account. And if I can run my master thesis simulation on the GPU.
  
  **Tags:** neural-networks, training, performance, hardware-evaluation
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2017-12-15. Score: 6 | Accepted answer
  
  Usually the problem is to fit the model into video RAM.
  If it does not, you cannot train your model at all without big efforts (like training parts of the model separately).
  If it does, time is your only problem. But the difference in training time between consumer GPUs like the Nvidia 1080 and much more expensive GPU accelerators like the Nvidia K80 are not very large. Actually the best consumer cards are faster than GPU accelerators, but lack other properties like VRAM. Random comparisons and benchmarks: [Which GPU(s) to Get for Deep Learning: My Experience and Advice for Using GPUs in Deep Learning](http://timdettmers.com/2017/04/09/which-gpu-for-deep-learning/), [A Comparison between NVIDIA’s GeForce GTX 1080 and Tesla P100 for Deep Learning](https://medium.com/@alexbaldo/a-comparison-between-nvidias-geforce-gtx-1080-and-tesla-p100-for-deep-learning-81a918d5b2c7).
  
  To calculate if your models fits into VRAM, you just approximate how much data and which hyperparameters you have (inputs, outputs, weights, layers, batch size, which datatype and so on).
  
  > **Comments:**
  > **username_0**: Good answer, but what kind of a model does not fit into a 6/8GB Memory? Let's say each batch of training data + the model. I have no idea how to calculate the size of the model but I cannot imagine a few thousands neurons and their weights can fill up Gigabytes of Memory. The Pretrained image recognition model from Google is a few hundred Megs https://www.tensorflow.org/tutorials/image_recognition and that can recognize more than most others out there. Apple handles it on-chip in the iPhone so the model itself must be much smaller than the T-data. And that doesn't need to be all in RAM right?  
  > **username_3**: I am often working with NLP models, mostly multi-layer RNNs, and if the vocabulary size is large, even 8 GB might not be enough.  Remember that training usually needs more space than prediction (gradients take up space also).  
  
  ---
  
  ### Answer by username_1 on 2017-12-13. Score: 3
  
  As a caveat, I’d suggest that unless you’re pushing up against fundamental technological limits, computation speed and resources should be secondary to design rationale when developing a neural network architecture.
  
  That said, earlier this year I finished my MS thesis that involved bioinformatics analytics pipelines with whole genome sequencing data - that project took over 100,000 hours of compute time to develop according to our clusters job manager. When your on a deadline, resources can be a real constraint and speed can be critical.
  
  So, to answer your questions as I understand them:
  
  **Would I have been better off to use the money to buy time in the cloud?**
  
  Probably. The few hundred dollars you spent on the 1060 would take you far training your model(s) in the cloud. Further, as far as I can tell, you don’t require the GPU to be cranking 100% of the time (you would if you were, say, mining crypto currencies). Finally, with cloud instances you could scale, training multiple models at once, which can speed up the exploration and validation of any architecture you settle on.
  
  **Is there a way to gauge the compute time of a neural network on a given GPU**
  
  Well, [Big O](https://en.m.wikipedia.org/wiki/Big_O_notation) is one estimator, but it sounds like you want a more precise method. I’m sure they exist, but I’d counter that you can make your estimation with simple back of the envelope calculations that account for threads, memory, code iterations, etc. Do you really want to dig into the GPU processing pipeline on the 1060? You may be able to come up with a very good estimate by understanding everything happening between your code and the metal, but ultimately it’s probably not worth the time and effort; it will likely confirm that Big O notation (the simple model, if you will) captures most of the variation in compute time. One thing you can do if you notice bottlenecks is performance profiling.
  
  
  ---
  
  ### Answer by username_2 on 2017-12-13. Score: 2
  
  It depends on what you need. You can train any size of network on any resource. The problem is the time of training. If you want to train Inception on an average CPU it will take months to converge. So, it all depends on how long you can wait to see your results based on your network. As in neural nets we do not have only one operation but many (like concatenating, max pooling, padding etc.), it is impossible to make an estimation as you are searching for. Just start training some infamous networks and measure the time. Then, you can interpolate how long it will take to train networks that you are searching for.
  
  > **Comments:**
  > **username_0**: Are you aware of a resource that collects information for different NN jobs? So a list of: NN type, unit count, hyperparameters, dataset size, Hardware used, time taken? That would help to develop a rule of thumb intuition  
  
  ---
  
  