Hey @Luca_Mingarelli
torch.distributed package can help to distribute tensors across multiple nodes, but, as of today, you still need to implement the distribute matrix multiplication on your own.  If you are looking for features like Mesh-TensorFlow. It is not yet available.
This is a very interesting request. Do you mind share some some details of the application?