Solved by Xiaopeng_Li in post #4 
              
              
                Close this issue because similar issue is already discussed in pull-19577 and pull-21736, and the PyTorch team have worked out the solution, that is using no_sync() context manager.