Hey Folks,
I am running my model on multiple gpus.And I have a tensor, which will be present on each Gpus, which I want to access. Now, I am looking to get hold of all these tensor, on all the gpus, and do some operation, in a synchronous fashion, and then broadcast the result on all the gpus, to be used in the next step.
For Example: the tensor we are talking is T, with T being present in all cuda 0-3(for 4 gpus). Now, I need to get hold of this T tensor(which has different values at different gpu) and get some stat out of this, and then send back this stat to all gpus.
Please suggest me how this can be achieved.