I found a problem with trying to run a high volume of images on a pipeline that includes pytorch and is supposed to run in a distributed way.  The dataset is over a million samples. The pipeline is having trouble serializing and deserializing very large objects. The training dataset (trn_ds in the code) is very large at 2.5 million images. I am using a batch process for processing the images from the data loader, but the data loader does rely upon the training dataset which holds all the image tensors.  One thing to try I thought of is to change the .pt (pytorch tensors) to numpy tensors.  Numpy tensors are apparently easier to serialize and deserialize.  But I’m having trouble finding documentation on how to create numpy image tensors in a dataset and then feed them to a data loader.  Is this possible?