I would guess your initial loss might be a bit higher using this new normalization approach, as your model would get “new” samples. Fine tuning the model might work, but again I would guess that you might get better results if you fine tune the entire model (not only the last layer(s)). These are just my guesses, so I would recommend to run some experiments and compare both approaches (and please share the results if you plan on running these tests!  ).