hm may the problem actually be that libtorch does not know onto which gpu to move the tensor? If that is the case, you could move your tensors to a certain gpu like so
module->to(torch::Device("cuda:0"));

where all following gpus are indexed in incrementing order, cuda:1, cuda:2 and so on