Thanks, @pritamdamania87,
Yes, I am using the DDP, and setting broadcast_buffers=False worked for me. Now I am wondering if it can affect any training outputs/accuracies?
Also, all the code is available at the main branch of GitHub - mmaaz60/mdetr. You just need to set requires_grad=True at mdetr/swin_transformers.py at fe1394c67e76a6c7e521bbda77d8294714038a3a · mmaaz60/mdetr · GitHub for reproducing the issue. I know it is a big codebase but I don’t have any shorter version for reproducing this behavior.