Thanks !!
By the way, is there any suggestions about finetune like this ? I plan to fine tune for 5 epochs, with the lr value and its annealing strategy not changed (I used cosine lr to train my model before fuse).  Shall I use a smaller lr or have the lr annealing curve changed ?