Solved by albanD in post #6 
              
              
                I am not sure this is true. 
You don’t use detach or no grad so the gradients will be properly tracked through your convmatrix2d function (even though the output matrix does not share its memory with the input, if you backprop, the gradients will flow back all the way to the input.