Thanks for your suggestion. When setting find_unused_parameters=True, the following warning was returned
Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters, consider turning this flag off. Note that this warning may be a false positive your model has flow control causing later iterations to have unused parameters. (function operator())
I think the problem is that parameters used in each trainer changes from time to time. It may happen that in some batches some parameters don’t appear in the computational graph due to random graph sampling, and it also can happen that every parameter appears in all trainers if sampling is balanced.