Solved by albanD in post #6 
              
              
                I guess what you want is: 
# Compute your loss as usual
out = model(input)
loss = loss_module(out, target)

# zero all gradients and backward
optimizer.zero_grad()
loss.backward()

# reset the gradients for the loss parameter
loss_module.zero_grad()

# compute the gradients for your loss parameters …