Body: <p>Training neural networks takes a while. My question is, how efficient is a neural network that is completely trained (assuming it's not a model that is constantly learning)?</p>
  <p>I understand that this is a vague and simply difficult question to answer, so let me be more specific: Imagine we have a trained Deep Neural Net, and even to be more specific it's a GPT-3 model.</p>
  <p>Now, we put the whole thing on a Raspberry Pi. No internet access. The whole process takes place locally.</p>
  <ul>
  <li><p>Will it run at all? Will it have enough RAM?</p>
  </li>
  <li><p>Now let's say we give it some text to analyze. Then we ask it a question. Will it take milliseconds to answer? Or is it going to be in the seconds? Minutes?</p>
  </li>
  </ul>
  <p>What I'm trying to understand, once a model is trained is it fairly performant because it's just essentially a bunch of very simple function calls on top of each other, or is it very heavy to execute? (perhaps due to the sheer number of these simple function calls)</p>
  <p>Please correct any misunderstanding about how the whole process works if you spot any. Thank you.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-04T14:26:44.760Z
Comments.Text: There's this article detailing why some models aren't suitable for practical applications: https://towardsdatascience.com/too-big-to-deploy-how-gpt-2-is-breaking-production-63ab29f0897c
Comments.UserDisplayName: Anton
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-04T15:03:56.353Z
Comments.Text: It seems to me that you're asking how [**efficient**](https://en.wikipedia.org/wiki/Algorithmic_efficiency) using a trained model is. Performance refers to how good the answers are in terms of correctness. So, I suggest that you edit your post to clarify this.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-04T15:43:32.343Z
Comments.Text: The GPT-3 model specifically is a monster, it requires powerful machine with over 350GB of RAM just to run inferences (and probably mulit-core processors too). There are plenty of others which are smaller, and likely there will be some that could run on a Rapberry Pi. So to prevent the answer being "it depends", could you clarify your use case? Sadly, there is no general answer here, and also if you want to run GPT-3 on any consumer hardware you are out of luck for a few years yet
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-04T16:41:32.647Z
Comments.Text: @NeilSlater Your comment answers my question. We wrote a custom albeit simple NLP processor and were wondering if we made the wrong decision instead of using GPT-3. Your answer clarifies that we didn't, because we would not be able to run such a model with our resources.
Comments.UserDisplayName: Anton
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-04T17:31:08.047Z
Comments.Text: @Anton: No problem. There are smaller versions of GPT-3 mind . . . I quoted stats for the largest one, which also has the best accuracy. I looked it up after the comment and saw there were options to use smaller variants that use the same approach but are less accurate. I don't know if the smallest one would fit on a Raspberry Pi, or would be useful for your project.
Comments.UserDisplayName: Neil Slater
Tags: <neural-networks><gpt><efficiency><computational-complexity><benchmarks>
Title: What is the efficiency of trained neural networks?
UserDisplayName: Anton
ThreadText: # What is the efficiency of trained neural networks?
  
  (Asked by: username_0 on 2020-09-04)
  
  Training neural networks takes a while. My question is, how efficient is a neural network that is completely trained (assuming it's not a model that is constantly learning)?
  
  I understand that this is a vague and simply difficult question to answer, so let me be more specific: Imagine we have a trained Deep Neural Net, and even to be more specific it's a GPT-3 model.
  
  Now, we put the whole thing on a Raspberry Pi. No internet access. The whole process takes place locally.
  
  * Will it run at all? Will it have enough RAM?
  * Now let's say we give it some text to analyze. Then we ask it a question. Will it take milliseconds to answer? Or is it going to be in the seconds? Minutes?
  
  What I'm trying to understand, once a model is trained is it fairly performant because it's just essentially a bunch of very simple function calls on top of each other, or is it very heavy to execute? (perhaps due to the sheer number of these simple function calls)
  
  Please correct any misunderstanding about how the whole process works if you spot any. Thank you.
  
  **Tags:** neural-networks, gpt, efficiency, computational-complexity, benchmarks
  > **Comments:**
  > **username_0**: There's this article detailing why some models aren't suitable for practical applications: https://towardsdatascience.com/too-big-to-deploy-how-gpt-2-is-breaking-production-63ab29f0897c  
  > **username_1**: It seems to me that you're asking how [**efficient**](https://en.wikipedia.org/wiki/Algorithmic_efficiency) using a trained model is. Performance refers to how good the answers are in terms of correctness. So, I suggest that you edit your post to clarify this.  
  > **username_2**: The GPT-3 model specifically is a monster, it requires powerful machine with over 350GB of RAM just to run inferences (and probably mulit-core processors too). There are plenty of others which are smaller, and likely there will be some that could run on a Rapberry Pi. So to prevent the answer being "it depends", could you clarify your use case? Sadly, there is no general answer here, and also if you want to run GPT-3 on any consumer hardware you are out of luck for a few years yet  
  > **username_0**: @NeilSlater Your comment answers my question. We wrote a custom albeit simple NLP processor and were wondering if we made the wrong decision instead of using GPT-3. Your answer clarifies that we didn't, because we would not be able to run such a model with our resources.  
  > **username_2**: @Anton: No problem. There are smaller versions of GPT-3 mind . . . I quoted stats for the largest one, which also has the best accuracy. I looked it up after the comment and saw there were options to use smaller variants that use the same approach but are less accurate. I don't know if the smallest one would fit on a Raspberry Pi, or would be useful for your project.  
  
  ---
  
  ## 0 answers
  
  