Scaling should make a difference.
e.g., consider the trivial scenario where L = (x-y)^2 vs. L= 5(x-y)^2.
dL/dx = 2(x-y) in the first and dL/dx = 10(x-y) in the second.
There is nothing wrong with the second approach, and there are many variations such as freezing part of the model trained on another method and finetuning a different part with a different objective. Of course, experimentation is probably the way to confirm what works and what doesn’t work well.
The tool of weighting losses can seem crude, but last I checked even huge models with countless objectives and sub-tasks (e.g., Tesla Autopilot) often balance the objectives in this way, often with hand-tuned weighting.