When using Pytorch to train a regression model with  very large dataset  (200*200*2200 image size and 10000 images in total) I found that the  system memory  (not GPU memory) grew during one epoch and finally the total system memory reached the size of all dataset,  as if all data were loaded into system memory.  (I have used DataLoader to generate data in batch and transfer the data to cuda device).
I thought after loading one batch of data the CPU would release the system memory, but it’s not the case. Can anyone tell me the how to fix this problem? The dataset is really too large and I only have limited system memory…