Solved by ptrblck in post #2 
              
              
                torch.unfold or nn.Unfold would propagate the gradients. 
Could you post your custom implementation of the conv module, so that we could have a look?