Let’s say I have a dataset which corresponds to some numpy array “data” already in memory.
If I use a DataLoader on top of this dataset to generate batches of size 1000, it seems the dataloader will call the method “getitem” 1000 times and cat the individual items together to create the batch.
However if I do it manually, I can directly access data[k : (k+1) * 1000], which is faster than getting the items individually.
Is there a way to customize the batch generation of the data loader so that I can optimize the batch generation in the case of a numpy array in memory?
thanks