Hello @pritamdamania87 , thanks for your reply, yes I’m aware of DDP being better than DP, but both have the exact same issue.
I added a DDP example in my post, and here’s more code that I’ve tried and resulted in the same issues:

  
      

      GitHub
  

  
    

yangkky/distributed_tutorial 6

  Contribute to yangkky/distributed_tutorial development by creating an account on GitHub.