Solved by Llama2 in post #4 
              
              
                I realized that I used a wrong loss function (manually implemented dice loss), which could have been undifferentiable and autograd might not have been able to backprop the loss (just a theory). I used nn.BCELoss() instead and it works now.