Hello again!
After more testing I found out that the problem is in the autoscaling of the losses. If I only autoscale the forward pass, everything works fine (of course, minus the proper loss handling). After diagnosing the losses, I also found that SSIM caused a similar behavior, but I think that the important thing is that while the model output is converted into a HalfTensor, the target (the ground truth image) is not, it stays as a FloatTensor.
Update: I fiddled around with the targets and that’s not the issue. They types are inconsistent, but it doesn’t change anything if they are the same type.