whilst looking at output.grad_fn.next_functions on autograd
I do see some “Addbackward” gradient functions which is presumably residual connections (and even though it is not, I do think we should keep the channel dimensions for these tensors since we are adding. plz point me out if it is wrong)
however, it seems this algorithm is quite daunting.
My colleague who is highly skilled had quite a trouble making an algorithm to find these residual connections and it only works for resnet. (tested on mobilenet, efficientnet (timm), and so on)
I would love to see if there are any ideas or code implemented for this.
Seems like knowing where the residual connection exists in the first place is the best attempt for this