Suppose I have 8 GPUs and each running some operations that produce different outputs. After aggregating, I would like to modify a variable self.x with this aggregated results. From what I understand if I write self.x = aggregate, all 8 processes will run this line once, which means in total 8 times. Is that right? If it is, how do I control this to be run only in the slowest process, so that it is only run once.
*It probably doesn’t affect the result, but it just feels “ugly” to me. I am not sure if it is normal for this kind of code to be written when writing distributed code. Hopefully someone can advise.