Solved by cs123951 in post #10 
              
              
                Hello, I have figured out what mistakes I take! 
I make mistakes in the init() of the model like this: 
mp_model = ToyMpModel(devices_list).cuda(devices_list[0])
ddp_mp_model = DDP(mp_model, device_ids=[0], find_unused_parameters=True)

I used multi-GPU model, but when I created model, I gave the mo…