This is quite strange, as we don’t expect CPU → GPU device change with no model/training side changes to incur unused parameters. This implied there are unused parameters on the GPU.
If you’re curious to dig into it, you can send find_unused=False and TORCH_DISTRIBUTED_DEBUG=DETAIL (need PyTorch 1.9) which will log the unused parameter name in the crash. Then you can check whether this parameter is being used or unused as part of DDP on CPU.