I am running into an issue where I have a custom convolutional layer using nn.Unfold, which I get the same values comparing with a regular nn.Conv2d 1 with the same weights initialized.
However, if I replace the layer when training a network, the weights seem not to be optimized as the regular nn.Conv2d do, causing the network to be unable to learn.
Is there any additional step to implement in order to backpropagate the error the same way as the nn.Conv2d does?
Thank you for your attention!