@KaiHoo What is the operation you want to perform across these 8 128x128 matrices? Allreduce will ensure that each GPU ends up with the average (or sum) of the matrices on all the GPUs. There are numerous such collective operations that you can perform to communicate data across GPUs in the distributed package that may be useful (docs here: https://pytorch.org/docs/stable/distributed.html#synchronous-and-asynchronous-collective-operations 4)