torch.from_numpy should be very fast - it only allocates a new torch.Tensor object that will reuse the same memory as the numpy array did. It’s nearly free  I’m surprised that it makes any difference for you