The placement of the gradient calculation and the optimizer calls look correct.
I still can’t find any issues. As the next debugging steps you could use a small subset of the original data and see, if this loss behavior is still visible in each epoch.



 iamneerav:

Can you tell me what to use instead of Variable if it is decrepated or maybe a link?


You can use tensors without wrapping them into Variables instead.
inputs = torch.autograd.Variable(train_data[0], requires_grad=False).to(device).permute(0,4,1,2,3)

would be now:
inputs = train_data[0].to(device).permute(0,4,1,2,3)