The difference in the loss reduction would create different loss ranges and thus also impact the gradient magnitudes, so you could also play around with the learning rate and try to find a sweet spot.