Hi,
The simple way I see to do this is the following:
You will have to first send all these Tensors to a common GPU. agregate the restults and compute your update. Then send the result back to each GPU.