Body: <p>I've been trying to figure out how to compute the number of Flops in backward pass of ResNet. For forward pass, it seems straightforward: apply the conv filters to the input for each layer. But how does one do the Flops counts for gradient computation and update of all weights during the backward pass?</p>
  
  <p>Specifically, </p>
  
  <ul>
  <li><p>how to compute Flops in gradient computations for each layer? </p></li>
  <li><p>what all gradients need to be computed so Flops for each of those can be counted? </p></li>
  <li><p>How many Flops in computation of gradient for Pool, BatchNorm, and Relu layers?</p></li>
  </ul>
  
  <p>I understand the chain rule for gradient computation, but having a hard time formulating how it'd apply to weight filters in conv layers of ResNet and how many Flops each of those would take. It'd be very useful to get any comments about method to compute total Flops for Backward pass. Thanks</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-05T10:56:08.397Z
Comments.Text: Hi and welcome to AI SE! Please, ask only one question. You're asking multiple questions. I suggest you edit your post to ask a single question and please focus on the smallest problem you need to solve before going to the next one.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-05T13:55:55.320Z
Comments.Text: It's actually just 1 question, how to get total op count without coding in backprop for a layer. I didn't get any answer so I thought I should at least lay the whole issue out for anyone to comment.
Comments.UserDisplayName: Joe Black
Tags: <neural-networks><deep-learning><convolutional-neural-networks><backpropagation><gradient-descent>
Title: Backward pass of CNN like Resnet: how to manually compute flops during backprop?
UserDisplayName: Joe Black
ThreadText: # Backward pass of CNN like Resnet: how to manually compute flops during backprop?
  
  (Asked by: username_0 on 2020-05-05)
  
  I've been trying to figure out how to compute the number of Flops in backward pass of ResNet. For forward pass, it seems straightforward: apply the conv filters to the input for each layer. But how does one do the Flops counts for gradient computation and update of all weights during the backward pass?
  
  Specifically,
  
  * how to compute Flops in gradient computations for each layer?
  * what all gradients need to be computed so Flops for each of those can be counted?
  * How many Flops in computation of gradient for Pool, BatchNorm, and Relu layers?
  
  I understand the chain rule for gradient computation, but having a hard time formulating how it'd apply to weight filters in conv layers of ResNet and how many Flops each of those would take. It'd be very useful to get any comments about method to compute total Flops for Backward pass. Thanks
  
  **Tags:** neural-networks, deep-learning, convolutional-neural-networks, backpropagation, gradient-descent
  > **Comments:**
  > **username_1**: Hi and welcome to AI SE! Please, ask only one question. You're asking multiple questions. I suggest you edit your post to ask a single question and please focus on the smallest problem you need to solve before going to the next one.  
  > **username_0**: It's actually just 1 question, how to get total op count without coding in backprop for a layer. I didn't get any answer so I thought I should at least lay the whole issue out for anyone to comment.  
  
  ---
  
  ## 0 answers
  
  