Answers.Body: <p>The graph seems to show the relative compute time of each component of the learning process, independent of the layer, using the typical meaning of layer in the context of artificial networks.</p>
  
  <p>The data is not unbelievable, in that one would expect a matrix-vector multiplication of an entire layer output would take more time than a matrix-vector multiplication of the size of a convolution kernel. Still, these representations are for a specific test condition and measurement methodology.</p>
  
  <p>Most of the theoretical and practical limits of parallel processing are heavily biased by seventy years of sequential processing as the foundation of computer science study and practice. If an understanding of parallel processing is desired, the von Neumann architecture is not the place to look. At the time of that development, computers could either be made of wood, steel, vacuum tubes, or relays. Developing a computer with 65536 parallel data paths was unrealistic to the extreme. The invention of algorithms was a workaround to funnel all processing through a single CPU. We've been backing out of that mind set for the last thirty years, beginning with FPUs, GPUs, multiple buses, computer clusters, and multiple cores. More parallelism in VLSI form is coming, and soon.</p>
  
  <p>The demonstration of parallel processing comes, as many things do, from biology. Look at farming and grain growth under the sun. Look at visual circuitry coming from the eyes of thousands of species. Consider the orbits of millions of planets and their satellites. The artificial parallelism to examine is the transportation systems and the Internet. Parallel processing is efficient when the process was designed as a parallel process rather than trying to make parallel that which was designed as serial. For instance, the human ear does the physical equivalent of a real time windowing FFT without performing a single multiplication operation.</p>
  
  <p>The bottleneck of the floating point multiplication of an attenuation matrix with a vector (parameterizing the mix of signals from one layer to the next) is a key to why NVidia has done well with re-purposing rendering hardware for artificial network training. It is also one of the many reasons this question is interesting: <a href="https://ai.stackexchange.com/questions/7328/if-digital-values-are-mere-estimates-why-not-return-to-analog-for-ai">If digital values are mere estimates, why not return to analog for AI?</a>.</p>
  
  <p>When surfing for GFLOPs vs compute time for different types of layers across GPUs and CPUs, using the term 'operation' instead of 'layer' in the search terms may help. Even though elementary operations are lower level than that of the comparison desired, the term layer is delegated strongly to the higher level meaning in the machine learning literature. Reliable data in this space is thin because much of the speed optimization has already been researched and developed, so whatever the visualization shows is useless unless you work on VLSI development at Intel, Samsung, NVidia, Qualcom, TI, or one of the other silicon giants.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-11T14:32:54.347Z
Answers.LastActivityDate: 2018-12-11T14:32:54.347Z
Answers.UserDisplayName: Douglas Daseeco
Body: <p><a href="https://i.sstatic.net/C2Wkg.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/C2Wkg.png" alt="enter image description here"></a></p>
  
  <p>Hello,
  I would like to know whether this picture from the paper: Distributed Training of Deep Neural Networks: Theoretical and Practical Limits of Parallel Scalability valid?</p>
  
  <p>Questions:<br>
  1) Does InnerProduct (Fully connected) layer actually take <strong>more time to compute in</strong> a neural network than Convolution? 
  <br>
  <br>
  2) Is assessing <strong>GLOPs/time</strong> a good way of <strong>estimating performance</strong> of different types of layers in a neural network on any hardware? (Conv, FC etc.)
  <br>
  <br>
  3) Does anyone know where I can find GFLOPs vs compute time for different types of layers across GPUs/CPUs? (I know DeepBench, any other suggestions would be great too)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-06-04T20:22:49.063Z
Comments.Text: Welcome to SE:AI!  (Thanks for cutting out the graphics and including them in the post:)
Comments.UserDisplayName: DukeZhou
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-08T20:42:16.623Z
Comments.Text: I think what you suggest requires an end to end revamp of the hardware architectures we have developed. I find it more feasible personally, to emulate these processes. Once we know the mechanism, we can make it an equation. Just a thought. But I agree with you, backpropogation and batching dont seem like 'intelligence'.
Comments.UserDisplayName: Mathbreaker
Tags: <neural-networks><convolutional-neural-networks><ai-design><algorithm><performance>
Title: Relative compute time for each type of layer in a neural network
UserDisplayName: Mathbreaker
ThreadText: # Relative compute time for each type of layer in a neural network
  
  (Asked by: username_0 on 2018-06-03)
  
  [![enter image description here](https://i.sstatic.net/C2Wkg.png)](https://i.sstatic.net/C2Wkg.png)
  
  Hello,
  I would like to know whether this picture from the paper: Distributed Training of Deep Neural Networks: Theoretical and Practical Limits of Parallel Scalability valid?
  
  Questions:  
  1) Does InnerProduct (Fully connected) layer actually take **more time to compute in** a neural network than Convolution?
    
    
  2) Is assessing **GLOPs/time** a good way of **estimating performance** of different types of layers in a neural network on any hardware? (Conv, FC etc.)
    
    
  3) Does anyone know where I can find GFLOPs vs compute time for different types of layers across GPUs/CPUs? (I know DeepBench, any other suggestions would be great too)
  
  **Tags:** neural-networks, convolutional-neural-networks, ai-design, algorithm, performance
  
  > **Comments:**
  > **username_1**: Welcome to SE:AI!  (Thanks for cutting out the graphics and including them in the post:)  
  > **username_0**: I think what you suggest requires an end to end revamp of the hardware architectures we have developed. I find it more feasible personally, to emulate these processes. Once we know the mechanism, we can make it an equation. Just a thought. But I agree with you, backpropogation and batching dont seem like 'intelligence'.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-12-11. Score: 2 | Accepted answer
  
  The graph seems to show the relative compute time of each component of the learning process, independent of the layer, using the typical meaning of layer in the context of artificial networks.
  
  The data is not unbelievable, in that one would expect a matrix-vector multiplication of an entire layer output would take more time than a matrix-vector multiplication of the size of a convolution kernel. Still, these representations are for a specific test condition and measurement methodology.
  
  Most of the theoretical and practical limits of parallel processing are heavily biased by seventy years of sequential processing as the foundation of computer science study and practice. If an understanding of parallel processing is desired, the von Neumann architecture is not the place to look. At the time of that development, computers could either be made of wood, steel, vacuum tubes, or relays. Developing a computer with 65536 parallel data paths was unrealistic to the extreme. The invention of algorithms was a workaround to funnel all processing through a single CPU. We've been backing out of that mind set for the last thirty years, beginning with FPUs, GPUs, multiple buses, computer clusters, and multiple cores. More parallelism in VLSI form is coming, and soon.
  
  The demonstration of parallel processing comes, as many things do, from biology. Look at farming and grain growth under the sun. Look at visual circuitry coming from the eyes of thousands of species. Consider the orbits of millions of planets and their satellites. The artificial parallelism to examine is the transportation systems and the Internet. Parallel processing is efficient when the process was designed as a parallel process rather than trying to make parallel that which was designed as serial. For instance, the human ear does the physical equivalent of a real time windowing FFT without performing a single multiplication operation.
  
  The bottleneck of the floating point multiplication of an attenuation matrix with a vector (parameterizing the mix of signals from one layer to the next) is a key to why NVidia has done well with re-purposing rendering hardware for artificial network training. It is also one of the many reasons this question is interesting: [If digital values are mere estimates, why not return to analog for AI?](https://ai.stackexchange.com/questions/7328/if-digital-values-are-mere-estimates-why-not-return-to-analog-for-ai).
  
  When surfing for GFLOPs vs compute time for different types of layers across GPUs and CPUs, using the term 'operation' instead of 'layer' in the search terms may help. Even though elementary operations are lower level than that of the comparison desired, the term layer is delegated strongly to the higher level meaning in the machine learning literature. Reliable data in this space is thin because much of the speed optimization has already been researched and developed, so whatever the visualization shows is useless unless you work on VLSI development at Intel, Samsung, NVidia, Qualcom, TI, or one of the other silicon giants.
  
  
  ---
  
  