Thanks @ptrblck, my current workaround is just as you suggested. I wonder if it’s possible to implement it in a more “elegant” way, i.e. we deal the case inside MyModel, such that the interface to upper invoking could be like as usual.
I also tried overwriting the state_dict and load_state_dict of the model. But it seems these functions is never used if we wrapped the model with DDP or DP.