Awesome example! clear and simple.  Thanks for it Howard! 
One follow up question, what is the best way to double check the distributed training actually happened.  Just want to make sure  I am not cheated by some lucky succeeded run of mis-configuration      I read this page (Distributed communication package - torch.distributed — PyTorch 1.8.1 documentation 7) but not sure what would be the definitive way to tell.