I am trying to write a Dataset of image patches (50x50 in HxW) from many (100k) large images (3Kx3K in HxW).
In the current implementation, __getitem__ function reads a image and randomly extract small patch. The overhead of loading and decoding makes the Dataset very slow.
Since I have too many large images, they can’t totally loaded into memory.
Is there any way to make the Dataset more efficient?
Thanks!