Thanks Can!  Really appreciate all the information!
My last question, I want to make sure the “parallel training” is indeed happening, would you please share a few code pointers where processes do the parameters sharing with others. I’d like to print out the parameter (maybe before and after as well) to make sure I have things rolling correctly.