If you create a new tensor, you will detach the inputs from the computation graph.
The creation of your matrix tensor might be the issue here.
Could you try to rewrite the creation using torch.cat instead?
Let me know, if that helps or if we need to dig a bit deeper.