Answers.Body: <p>As CNN contains convolution operation, but DNN uses constructive divergence for training. CNN is more complex in terms of Big O notation.</p>
  
  <p>For reference:</p>
  
  <ol>
  <li><p>See <a href="https://arxiv.org/pdf/1412.1710.pdf" rel="nofollow noreferrer">Convolutional Neural Networks at Constrained Time Cost</a> for more details about the time complexity of CNNs</p></li>
  <li><p>See <a href="https://ai.stackexchange.com/q/13612/2444">What is the time complexity of the forward pass algorithm of a neural network?</a> and <a href="https://ai.stackexchange.com/q/5728/2444">What is the time complexity for training a neural network using back-propagation?</a> for more details about the time complexity of the forward and backward passes of an MLP</p></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-19T22:54:56.113Z
Answers.Text: Indeed, purely empirical experience tells me that fully connected layers train much faster, even with hugely more parameters.  The operations in them are simple, as is the backprop.  CNNs appear to use huge amounts of memory to store all the intermediate gradient terms.
Answers.UserDisplayName: Mastiff
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-08T07:12:46.867Z
Answers.LastActivityDate: 2020-05-19T19:46:49.420Z
Answers.LastEditDate: 2020-05-19T19:46:49.420Z
Answers.UserDisplayName: ketul parikh
Body: <p>In a convolutional neural network, which layer consumes more training time: convolution layers or fully connected layers? </p>
  
  <p>We can take AlexNet architecture to understand this. I want to see the time breakup of the training process. I want a relative time comparison so we can take any constant GPU configuration.</p>
  
Tags: <neural-networks><deep-learning><convolutional-neural-networks><training>
Title: Which layer in a CNN consumes more training time: convolution layers or fully connected layers?
UserDisplayName: Ruchit Dalwadi
ThreadText: # Which layer in a CNN consumes more training time: convolution layers or fully connected layers?
  
  (Asked by: username_0 on 2018-09-06)
  
  In a convolutional neural network, which layer consumes more training time: convolution layers or fully connected layers?
  
  We can take AlexNet architecture to understand this. I want to see the time breakup of the training process. I want a relative time comparison so we can take any constant GPU configuration.
  
  **Tags:** neural-networks, deep-learning, convolutional-neural-networks, training
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-09-08. Score: 5
  
  As CNN contains convolution operation, but DNN uses constructive divergence for training. CNN is more complex in terms of Big O notation.
  
  For reference:
  
  1. See [Convolutional Neural Networks at Constrained Time Cost](https://arxiv.org/pdf/1412.1710.pdf) for more details about the time complexity of CNNs
  2. See [What is the time complexity of the forward pass algorithm of a neural network?](https://ai.stackexchange.com/q/13612/2444) and [What is the time complexity for training a neural network using back-propagation?](https://ai.stackexchange.com/q/5728/2444) for more details about the time complexity of the forward and backward passes of an MLP
  
  > **Comments:**
  > **username_2**: Indeed, purely empirical experience tells me that fully connected layers train much faster, even with hugely more parameters.  The operations in them are simple, as is the backprop.  CNNs appear to use huge amounts of memory to store all the intermediate gradient terms.  
  
  ---
  
  