Hi, one more question:
I have a huge tensor (700MB, precomputed, requires_grad=False) which is used for  tensor multiplication somewhere as a Module (as shown in the snippet)
When training the model with multiple GPUs, I need to push it to all GPUs. The easiest way would be using regist_buffer in a module. However this means the stat_dict would be larger than 700MB ( definitely not a good idea). So I was wondering the best way to push such a large tensor to all GPUs?
BTW, if I simply use “tensor.to(device)” , is the tensor gonna be pushed to all GPUs or only the default one?  (Had a test, seems like it is on the defalult gpu not all gpus.)
Thanks in advance!
class NewModule(nn.Module):

    def __init__(self, pre_matrix):
        super(NewModule, self).__init__()
        # Pre_matrix: NXP, of size 700MB, requires_grad=False
        self.pre_matrix = pre_matrix
        self.pre_matrix.requires_grad=False
        # self.register_buffer('pre_matrix', pre_matrix)  ### this means the stat_dic is larger than 700MB

    def forward(self, input):
        # input: MXN, on multiple gpus
        # output: MXP, on multiple gpus
        out = input @ self.Pre_matrix
        return out