I’m not sure if my case is what you want but I use GPU communication functions to plot loss graph during training.
I use a custom Loss class that inherits nn.modules.loss._Loss.
It calculates the loss and stores the record and plots the loss graph.
The loss values are synchronized inside the Loss class
Thus, I don’t have to send values to main() scope.
Here’s my public github code 4.
all_reduce function performs sync and it is called at the end of an epoch from trainer.