For me, simply setting find_unused_parameters to False in DistributedDataParallel (DDP) solves the problem. There is no error anymore.
Ref: https://stackoverflow.com/questions/68000761/pytorch-ddp-finding-the-cause-of-expected-to-mark-a-variable-ready-only-once 27