Thank you, seungjun
I appreciate your code.
I understand that all_reduce function transmits loss between GPUs.
I am trying to evaluate performance of optimizers by changing many factors such as batch size and learning rate. Therefore, I have multiple loops to change the variables outside the optimization loop. Also, I have to collect many kinds of data along with loss and accuracy to analyze the details of the optimizers and plot them.
I felt a kind of odd about performing such non-multiplication looping tasks and all plotting using expensive GPU and its memory. So, I tried to implement all the outer loops, analysis, and plotting tasks in main(), which requires getting the information from the spawned GPU processes.
Thank you.