My last question, I want to make sure the “parallel training” is indeed happening, would you please share a few code pointers where processes do the parameters sharing with others.

For clarification, the architecture is multiple nodes and multiple processes within a node, with one process doing the training? Also, are you using a DDP model?