I’m not sure how the training will behave, if the optimizer uses internal running estimate of some parameters while others are empty, so your best bet is to try it out and please report your findings.