Hello,
I have been trying to debug an issue where, when working with a dataset, my RAM is filling up quickly. It turns out this is caused by the transformations I am doing to the images, using transforms.
My code is very simple:
  for dir1 in os.listdir(img_folder):
      for file in os.listdir(os.path.join(img_folder, dir1)):
          image_path = os.path.join(img_folder, dir1,  file)
          with Image.open(image_path) as img_pil:
            normalize = transforms.Normalize(mean=mean,std=std)
            preprocess = transforms.Compose([
              transforms.Resize((img_size,img_size)),
              transforms.ToTensor(),
              normalize
            ])
            img_pil = preprocess(img_pil)

Without running the “preprocess code”, the memory is emptied correctly upon opening and closing images.
I have tried defining the normalize and preprocess function outside the loop, but memory was still accumulating.
Am I missing something? Is there a way to free up the memory that is being occupied by the transformation steps?
NB: same issue arises when using a dataloader. But I didn’t know what was causing it, that’s how I ended up here.
Thanks