Answers.Body: <p>These are some ways to speed up inference:</p>
  
  <ul>
  <li><p>Reduction of float precision: This is done post-training. According to work in this segment, a very little accuracy is sacrificed for a huge benefit in memory usage reduction. Also it speeds up inference. float32 -> float8.
  Reference paper: <a href="https://arxiv.org/pdf/1502.02551" rel="nofollow noreferrer">https://arxiv.org/pdf/1502.02551</a></p></li>
  <li><p>Using ReLU or similar small compute power activations: Benefits are obvious when you don't need to compute heavy exponents, like in tanh or sigmoid.</p></li>
  <li><p>Binary Neural Architecture: This is new. This taps upon the ability to use binary valued weights and activations (1 bit) compared to 32 bit float counterparts. Estimation and learning is done through POPCNT and XNOR operations (for matrix products) and STE (straight through estimator) for backpropagation. You do have to sacrifice for a large number of neurons to learn the same features, <em>but, the speed is on average 7x faster</em>. Reference paper: <a href="https://pjreddie.com/media/files/papers/xnor.pdf" rel="nofollow noreferrer">https://pjreddie.com/media/files/papers/xnor.pdf</a></p></li>
  </ul>
  
  <p><a href="https://software.intel.com/en-us/articles/binary-neural-networks" rel="nofollow noreferrer">https://software.intel.com/en-us/articles/binary-neural-networks</a></p>
  
  <ul>
  <li>Hardware standpoint: Using specialised hardware to compute matrix products, pretty standard examples are GPUs and TPUs.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-10T14:39:53.507Z
Answers.Text: Yeah, but they do seem promising
Answers.UserDisplayName: Karan Shah
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-01T02:08:07.263Z
Answers.LastActivityDate: 2019-05-08T18:04:22.470Z
Answers.LastEditDate: 2019-05-08T18:04:22.470Z
Answers.UserDisplayName: Karan Shah
Body: <p>One way to speed up a neural network is to prune the network and reducing number of neurons in each layer. What are the other methods to speed up inference?</p>
  
Tags: <neural-networks><deep-learning><convolutional-neural-networks>
Title: What are the various methods for speeding up neural network for inference?
UserDisplayName: Illuminati0x5B
ThreadText: # What are the various methods for speeding up neural network for inference?
  
  (Asked by: username_0 on 2019-04-30)
  
  One way to speed up a neural network is to prune the network and reducing number of neurons in each layer. What are the other methods to speed up inference?
  
  **Tags:** neural-networks, deep-learning, convolutional-neural-networks
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-05-01. Score: 2 | Accepted answer
  
  These are some ways to speed up inference:
  
  * Reduction of float precision: This is done post-training. According to work in this segment, a very little accuracy is sacrificed for a huge benefit in memory usage reduction. Also it speeds up inference. float32 -> float8.
    Reference paper: <https://arxiv.org/pdf/1502.02551>
  * Using ReLU or similar small compute power activations: Benefits are obvious when you don't need to compute heavy exponents, like in tanh or sigmoid.
  * Binary Neural Architecture: This is new. This taps upon the ability to use binary valued weights and activations (1 bit) compared to 32 bit float counterparts. Estimation and learning is done through POPCNT and XNOR operations (for matrix products) and STE (straight through estimator) for backpropagation. You do have to sacrifice for a large number of neurons to learn the same features, *but, the speed is on average 7x faster*. Reference paper: <https://pjreddie.com/media/files/papers/xnor.pdf>
  
  <https://software.intel.com/en-us/articles/binary-neural-networks>
  
  * Hardware standpoint: Using specialised hardware to compute matrix products, pretty standard examples are GPUs and TPUs.
  
  > **Comments:**
  > **username_1**: Yeah, but they do seem promising  
  
  ---
  
  