Well, update the

dataloders.sampler = sampler

is not necessary. Just to keep the sampler the same as batch_sampler.
I cannot find any elegant way to update the sampler and batch size, but the above way is not too bad ( 3 lines of code)
As the distribution of my training classes is unbalanced. Start sampling equally then gradually converge to final distribution seems working fine.