You could create a Dataset and load the data lazily.
However, if you have already loaded the numpy arrays, they should apparently fit into your RAM.
Try to use torch.from_numpy to reuse the underlying memory and to avoid a copy.