Answers.Body: <p>Regarding question 1:</p>
  <ul>
  <li>Use <strong>NAS</strong> when you don't know which is the best NN architecture for your problem. Basically, you start from a set of pre-defined blocks and NAS finds the best way to connect them and also their hyper-parameters like number of blocks, filters, etc.</li>
  <li>As you said, <strong>hyper-opt</strong> is useful for an already designed NN (i.e. you know that such an architecture should work well) but also for the optimizer (e.g. learning rate, weight decay), and other hyper-parameters related to some custom layers, operation, loss. For example, in auto-encoders the size of the latent space is another hyper-parameter.</li>
  <li>Finally, <strong>pruning</strong> is a <em>model compression</em> technique. So you use that to remove a given % of connections in your <em>trained</em> model: basically it sets some weights to zero. Pruning is done to get a lightweight version of your initial model, with the aim of deploying it on mobile, low-power devices and even FPGAs (field programmable gate arrays) boards. The pruned model has to be compiled by specific tools in order to get a practical advantage of the weights set to zero, to reduce the number of operations and also memory usage.</li>
  </ul>
  <p>About question 2:</p>
  <ul>
  <li>Well, I would use NAS first to determine the best NN architecture, then hyper-opt to find the best hyper-params of, e.g., the optimizer (and so not the ones related to the NN since NAS should have already found them), and finally, pruning as a post-training step but only if you want to deploy the model on some computing device with limited resources.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-15T07:37:11.840Z
Answers.LastActivityDate: 2023-04-15T07:37:11.840Z
Answers.UserDisplayName: Luca Anzalone
Body: <p><strong>As far as I understand:</strong></p>
  <ol>
  <li><strong>NAS</strong> = an algorithm that searches for the best NN architecture, i.e., how many layers, what activation function to use, how many neurons, etc.</li>
  <li><strong>Hyperparameter Optim</strong> = finding the best set of values for an already designed NN model.</li>
  <li><strong>Pruning</strong> = optimizing the connections between layers in a trained model.</li>
  </ol>
  <p><strong>My question is,</strong></p>
  <ul>
  <li>When should I use which in the above?</li>
  <li>Can all three be used at a time?</li>
  </ul>
  
Tags: <neural-networks><hyperparameter-optimization>
Title: What is the basic difference between NAS, Hyperparameter Optimization/Tuning and Pruning?
UserDisplayName: user366312
ThreadText: # What is the basic difference between NAS, Hyperparameter Optimization/Tuning and Pruning?
  
  (Asked by: username_0 on 2023-04-14)
  
  **As far as I understand:**
  
  1. **NAS** = an algorithm that searches for the best NN architecture, i.e., how many layers, what activation function to use, how many neurons, etc.
  2. **Hyperparameter Optim** = finding the best set of values for an already designed NN model.
  3. **Pruning** = optimizing the connections between layers in a trained model.
  
  **My question is,**
  
  * When should I use which in the above?
  * Can all three be used at a time?
  
  **Tags:** neural-networks, hyperparameter-optimization
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-04-15. Score: 2 | Accepted answer
  
  Regarding question 1:
  
  * Use **NAS** when you don't know which is the best NN architecture for your problem. Basically, you start from a set of pre-defined blocks and NAS finds the best way to connect them and also their hyper-parameters like number of blocks, filters, etc.
  * As you said, **hyper-opt** is useful for an already designed NN (i.e. you know that such an architecture should work well) but also for the optimizer (e.g. learning rate, weight decay), and other hyper-parameters related to some custom layers, operation, loss. For example, in auto-encoders the size of the latent space is another hyper-parameter.
  * Finally, **pruning** is a *model compression* technique. So you use that to remove a given % of connections in your *trained* model: basically it sets some weights to zero. Pruning is done to get a lightweight version of your initial model, with the aim of deploying it on mobile, low-power devices and even FPGAs (field programmable gate arrays) boards. The pruned model has to be compiled by specific tools in order to get a practical advantage of the weights set to zero, to reduce the number of operations and also memory usage.
  
  About question 2:
  
  * Well, I would use NAS first to determine the best NN architecture, then hyper-opt to find the best hyper-params of, e.g., the optimizer (and so not the ones related to the NN since NAS should have already found them), and finally, pruning as a post-training step but only if you want to deploy the model on some computing device with limited resources.
  
  
  ---
  
  