Wouldn’t this slow down the training again and we have the original model back? 
Anyway, I still think it’s an interesting idea and some quantization approaches use exactly this strategy (losing a bit of model accuracy to gain speed).