Well when you concatenate the Tensor, you will end up with a single Tensor. That single Tensor can be only on a single device, So you already compute the loss on a single device no?
@ptrblck do we automatically send Tensors accross devices in torch.cat? That sounds wrong no?