[s2s] metrics.json is wrong on multigpu
overwritten by last rank to save it.

@nateraw is there a way to check if my module `is_rank_zero` or some such?
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
