Solved by ptrblck in post #2 
              
              
                If you scale the loss, you’ll also scale the gradients. 
In a simple use case, this can be used instead of changing the learning rate, as seen here: 
# Setup
torch.manual_seed(2809)

lin = nn.Linear(2, 2, bias=False)
x = torch.randn(1, 2)

# Standard approach
out = lin(x)
loss = out.sum()
print(loss…