When I add the loss weight or some new losses (keeping the original losses unchanged) to a model, I found the norm of gradient of the model will increase. For example, use the following code:
total_norm = 0.0
for p in net.parameters():
    param_norm = p.grad.data.norm(2)
    total_norm += param_norm.item() ** 2
    total_norm = total_norm ** (1. / 2)
print(total_norm)

I wonder if it will affect the model training stability or performance. Should I decrease the learning rate or scale the loss weight to match the original gradient norm? Does anyone can help clarify the relationship between gradient/norm of gradient vs learning rate vs step size? Thanks a lot!