marcel1991:

So it seems like somehow the gradients are not flowing properly back until the input.


You could check it by printing all .grad attributes of the model parameters after the backward call.
If some of them are None, your computation graph was (accidentally) detached at some point.



 marcel1991:

I am usin Pytorch 1.2.0 with Cuda 10.0


Could you update to the latest stable version and post a code snippet to reproduce this issue?



 marcel1991:

Or is it really a bug with pytorch/cuda?


My code snippet yields the same output and gradients for both models, so I assume it’s not a bug in PyTorch/CUDA but in the user code.