Answers.Body: <p><strong>Training and inference are usually completed on two separate systems</strong>. You are right in knowing that training of deep neural networks is usually done on GPUs and that inference is usually done on CPUs. However, training and inference are almost always done on two separate systems.</p>
  <p>The main workflow for many data scientists today is as follows:</p>
  <ol>
  <li><p>Create and establish all hyper-parameters for a model such as a deep neural network</p>
  </li>
  <li><p>Train the deep neural network using a GPU</p>
  </li>
  <li><p>Save the weights that training on the GPU established so that the model can be deployed.</p>
  </li>
  <li><p>Code the model in a production application with the optimal weights found in training.</p>
  </li>
  </ol>
  <p>So, as you can see from this workflow, training and inference are done in two completely separate phases.</p>
  <p>However, in some specific cases, training and inference are done on the same system. For example, if you are using a Deep Neural Network to play video games, then you may have the neural network train and infer on the same system. This would lead to more efficiency because it would allow the model to continuously learn.</p>
  <p>To answer your question on memory, the only applications where inference and training are done in the same application have a lot of memory available (think dual GPU dual CPU 128GB of RAM workstations), whereas applications that have a limited amount of memory only use inference such as embedded applications.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-03-07T02:30:29.543Z
Answers.Text: Thanks a lot, that was really helpful. But, wouldn't it be better for embedded applications such as autonomous cars to have training phase too? It's never too safe for a car to stop learning, I guess.
Answers.UserDisplayName: Mahmoud Abdel-Mon'em
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-03-07T04:13:46.720Z
Answers.Text: @MahmoudAbdel-Mon'em it actually would not be that safe for a car to keep learning while in production because it may pick up a bad learning pattern for example: if I drive recklessly I will be able to get to the destination faster and these patterns are not monitored. It would be better if the manufacturer periodically published.  a better model.
Answers.UserDisplayName: Aiden Grossman
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-03-07T12:55:18.373Z
Answers.Text: I take it that your answer is strictly applicable to the example of autonomous cars. What if we want something more generic? Let's assume an X application that uses DL, would some applications use both hardware systems on them? Or still they'll be using inference system with data pulled from a trained system away?
Answers.UserDisplayName: Mahmoud Abdel-Mon'em
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-03-07T14:49:53.717Z
Answers.Text: @MahmoudAbdel-Mon'em except for very very specialized applications you will not have inference and training done on the same system.
Answers.UserDisplayName: Aiden Grossman
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-03-07T16:14:43.527Z
Answers.Text: thank you. Can you please give me an example of some of the specialized applications that have both systems?
Answers.UserDisplayName: Mahmoud Abdel-Mon'em
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-03-07T01:58:44.713Z
Answers.LastActivityDate: 2021-12-19T13:48:31.127Z
Answers.LastEditDate: 2021-12-19T13:48:31.127Z
Answers.UserDisplayName: Aiden Grossman
Answers.Body: <p>Deep learning seems mostly to be a buzzword for what is essentially a neural network. You train with a data set to recognize a pattern, then input new data which is then classified by the trained network.</p>
  
  <p>So you train a neural network with 10 different kinds of animals using thousands of pictures. Then you show the network say 100 new images and have the network "guess"  what each animal is. </p>
  
  <p>The point here is that training a neural network would required code for feedback that an application using the trained network would not need. So an application just using the trained network would be a bit more streamlined than an application which could allow additional training data. </p>
  
  <p>What is missing is the ability for machine learning to form hierarchical or otherwise interrelated rules from the trained network so that the trained network is closer to being able to rationally explain why the classification works. Going back to the 10 animal trained network, without formulation of rules during training, there is no practical way for the network to reveal why any of the animals was classified the way it was. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-03-07T01:42:01.010Z
Answers.Text: Thanks for your reply, was helpful. But I still didn't know about the hardware usage part, can you elaborate on that more?
Answers.UserDisplayName: Mahmoud Abdel-Mon'em
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-03-06T17:47:42.840Z
Answers.LastActivityDate: 2018-10-22T03:43:47.680Z
Answers.LastEditDate: 2018-10-22T03:43:47.680Z
Answers.UserDisplayName: MaxW
Body: <p>From what I understand, there are 2 stages for deep learning: the first is training and the second is inference. The first is often done on GPUs because of their massive parallelism capabilities, among other things. The second, inference, while it can be done on GPUs, it's not used that much, because of power usage, and because the data presented while inferring are much less, so the full capabilities of GPUs won't be much needed. Instead, FPGAs and CPUs are often used for that.</p>
  <p>My understanding also is that a complete DL system will have both, a training system and an inferring system.</p>
  <p>My question is: are both systems required on the same application?</p>
  <p>Let's assume an autonomous car or an application where visual and image recognition is done, will it have both a training system to be trained and an inference system to execute? Or it has only the inference system and will communicate with a distant system that is already trained and has built a database?</p>
  <p>Also, if the application has both systems, will it have a big enough memory to store the training data? Given that it can be a small system and memory is ultimately limited.</p>
  
Tags: <deep-learning><training><implementation><inference>
Title: Are both the training and inference systems required in the same application?
UserDisplayName: Mahmoud Abdel-Mon'em
ThreadText: # Are both the training and inference systems required in the same application?
  
  (Asked by: username_0 on 2017-03-06)
  
  From what I understand, there are 2 stages for deep learning: the first is training and the second is inference. The first is often done on GPUs because of their massive parallelism capabilities, among other things. The second, inference, while it can be done on GPUs, it's not used that much, because of power usage, and because the data presented while inferring are much less, so the full capabilities of GPUs won't be much needed. Instead, FPGAs and CPUs are often used for that.
  
  My understanding also is that a complete DL system will have both, a training system and an inferring system.
  
  My question is: are both systems required on the same application?
  
  Let's assume an autonomous car or an application where visual and image recognition is done, will it have both a training system to be trained and an inference system to execute? Or it has only the inference system and will communicate with a distant system that is already trained and has built a database?
  
  Also, if the application has both systems, will it have a big enough memory to store the training data? Given that it can be a small system and memory is ultimately limited.
  
  **Tags:** deep-learning, training, implementation, inference
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2017-03-07. Score: 1 | Accepted answer
  
  **Training and inference are usually completed on two separate systems**. You are right in knowing that training of deep neural networks is usually done on GPUs and that inference is usually done on CPUs. However, training and inference are almost always done on two separate systems.
  
  The main workflow for many data scientists today is as follows:
  
  1. Create and establish all hyper-parameters for a model such as a deep neural network
  2. Train the deep neural network using a GPU
  3. Save the weights that training on the GPU established so that the model can be deployed.
  4. Code the model in a production application with the optimal weights found in training.
  
  So, as you can see from this workflow, training and inference are done in two completely separate phases.
  
  However, in some specific cases, training and inference are done on the same system. For example, if you are using a Deep Neural Network to play video games, then you may have the neural network train and infer on the same system. This would lead to more efficiency because it would allow the model to continuously learn.
  
  To answer your question on memory, the only applications where inference and training are done in the same application have a lot of memory available (think dual GPU dual CPU 128GB of RAM workstations), whereas applications that have a limited amount of memory only use inference such as embedded applications.
  
  > **Comments:**
  > **username_0**: Thanks a lot, that was really helpful. But, wouldn't it be better for embedded applications such as autonomous cars to have training phase too? It's never too safe for a car to stop learning, I guess.  
  > **username_1**: @MahmoudAbdel-Mon'em it actually would not be that safe for a car to keep learning while in production because it may pick up a bad learning pattern for example: if I drive recklessly I will be able to get to the destination faster and these patterns are not monitored. It would be better if the manufacturer periodically published.  a better model.  
  > **username_0**: I take it that your answer is strictly applicable to the example of autonomous cars. What if we want something more generic? Let's assume an X application that uses DL, would some applications use both hardware systems on them? Or still they'll be using inference system with data pulled from a trained system away?  
  > **username_1**: @MahmoudAbdel-Mon'em except for very very specialized applications you will not have inference and training done on the same system.  
  > **username_0**: thank you. Can you please give me an example of some of the specialized applications that have both systems?  
  
  ---
  
  ### Answer by username_2 on 2017-03-06. Score: 0
  
  Deep learning seems mostly to be a buzzword for what is essentially a neural network. You train with a data set to recognize a pattern, then input new data which is then classified by the trained network.
  
  So you train a neural network with 10 different kinds of animals using thousands of pictures. Then you show the network say 100 new images and have the network "guess" what each animal is.
  
  The point here is that training a neural network would required code for feedback that an application using the trained network would not need. So an application just using the trained network would be a bit more streamlined than an application which could allow additional training data.
  
  What is missing is the ability for machine learning to form hierarchical or otherwise interrelated rules from the trained network so that the trained network is closer to being able to rationally explain why the classification works. Going back to the 10 animal trained network, without formulation of rules during training, there is no practical way for the network to reveal why any of the animals was classified the way it was.
  
  > **Comments:**
  > **username_0**: Thanks for your reply, was helpful. But I still didn't know about the hardware usage part, can you elaborate on that more?  
  
  ---
  
  