Answers.Body: <p>You could look into <a href="https://www.tensorflow.org/model_optimization/guide/pruning" rel="nofollow noreferrer">models pruning</a>. There are several <a href="https://towardsdatascience.com/model-compression-via-pruning-ac9b730a7c7b" rel="nofollow noreferrer">techniques</a> out here, and all of them aim to reduce the amount of parameters of a model without affecting its performance metrics. Of course less parameters means less calculation and therefore faster inference time.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-12T22:54:23.817Z
Answers.Text: Thank you. I haven't looked into pruning much because it's not as straightforward as the others I've done. I'll look more into it now.
Answers.UserDisplayName: theastronomist
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-10T08:02:22.757Z
Answers.LastActivityDate: 2022-02-10T08:02:22.757Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>I am aware of several ways to optimize a convolutional (or any) model after training to make inferencing quicker. I am currently implementing <a href="https://scortex.io/batch-norm-folding-an-easy-way-to-improve-your-network-speed/" rel="nofollow noreferrer">BatchNormalization Folding</a> and removing Dropout layers from the network. I am also aware of <a href="https://www.tensorflow.org/lite/performance/post_training_float16_quant" rel="nofollow noreferrer">post training quantization</a> (specifically 16-bit quantization for use on GPU).</p>
  <p>Are there other layer optimization techniques that I can use other than quantization?</p>
  <p>My current model uses, Conv2D, Activation(relu), BatchNormalization, Dropout, Dense layers.</p>
  <p>Basic mnist Convnet metrics for 10K images, batch size of 1: (All have 98.92% accuracy)</p>
  <ul>
  <li>Original Network: 49.3s</li>
  <li>Folded Network: 33.37s</li>
  <li>Quantized Original: 7.449s</li>
  </ul>
  
Tags: <convolutional-neural-networks><tensorflow><optimization><batch-normalization>
Title: Additional Optimizations for Convolutional Models On Inferencing
UserDisplayName: theastronomist
ThreadText: # Additional Optimizations for Convolutional Models On Inferencing
  
  (Asked by: username_0 on 2022-02-10)
  
  I am aware of several ways to optimize a convolutional (or any) model after training to make inferencing quicker. I am currently implementing [BatchNormalization Folding](https://scortex.io/batch-norm-folding-an-easy-way-to-improve-your-network-speed/) and removing Dropout layers from the network. I am also aware of [post training quantization](https://www.tensorflow.org/lite/performance/post_training_float16_quant) (specifically 16-bit quantization for use on GPU).
  
  Are there other layer optimization techniques that I can use other than quantization?
  
  My current model uses, Conv2D, Activation(relu), BatchNormalization, Dropout, Dense layers.
  
  Basic mnist Convnet metrics for 10K images, batch size of 1: (All have 98.92% accuracy)
  
  * Original Network: 49.3s
  * Folded Network: 33.37s
  * Quantized Original: 7.449s
  
  **Tags:** convolutional-neural-networks, tensorflow, optimization, batch-normalization
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-02-10. Score: 2 | Accepted answer
  
  You could look into [models pruning](https://www.tensorflow.org/model_optimization/guide/pruning). There are several [techniques](https://towardsdatascience.com/model-compression-via-pruning-ac9b730a7c7b) out here, and all of them aim to reduce the amount of parameters of a model without affecting its performance metrics. Of course less parameters means less calculation and therefore faster inference time.
  
  > **Comments:**
  > **username_0**: Thank you. I haven't looked into pruning much because it's not as straightforward as the others I've done. I'll look more into it now.  
  
  ---
  
  