Thanks for your post! Are you using weight sharing in your module? Looking at does Gradient checkpointing support multi-gpu ? · Issue #63 · allenai/longformer · GitHub 2, it mentions that DDP does not work with gradient checkpointing + weight sharing in some cases, but we would need a more detailed reproduction to confirm the issue.
If you can get a repro of the issue, it would be great to file an issue at Issues · pytorch/pytorch · GitHub 2 so we can look into it.