I will try to check it when I have time if the gradients are flowwing back properly. Unfortunately I am not able to update Cuda 10.0 that easily and Pytorch 1.2. is the latest version for Cuda 10.0. I already updated Pytorch from 1.1 to 1.2 but same behaviour in both versions.
This behaviour just happens with the same model that is forwarded twice with two different inputs and where both outputs are somehow influencing each other.
E.g. something like this:
output1 = model.forward(input1)
output2 = model.forward(input2)

loss = (output1 - output2).pow(2).sum()

But where the optimal output somehow can only be achieved by considering the input and not just by setting the output all to zero, like in this example. I try to think of a simple reproducable example because I cannot post my code here and also not reduce my code which is a quite complex.