UPDATE:
It seems that the issue is worse than i thought. It could be related to any tensor operation. Simple operation such as changing the type of the tensor to float32 is causing this memory problem as well.
For some reason, the memory is not being cleaned.
PS: I tried forcing garbage collection. It was not useful.
          with Image.open(image_path) as img_pil:
            img_pil = torch.from_numpy(np.array(img_pil))
            img_pil = img_pil.type(torch.float32)