Is the result of p.requires_grad and p.grad is not None always the same across all process and all parameters? If allreduce ops on different processes could run into desync.
Which backend are you using (NCCL/Gloo/MPI) and which PyTorch version are you using? It will be helpful to have a min repro of this error.