Solved by eqy in post #2 
              
              
                Scaling should make a difference. 
e.g., consider the trivial scenario where L = (x-y)^2 vs. L= 5(x-y)^2. 
dL/dx = 2(x-y) in the first and dL/dx = 10(x-y) in the second. 
There is nothing wrong with the second approach, and there are many variations such as freezing part of the model trained on anot…