I comment these lines, and then everything works fine:
mask = coco.annToMask(elem)
mask = cv2.resize(mask, (self.w, self.h),
                              interpolation=cv2.INTER_AREA)
elem['mask'] = mask

I guess it’s because the originally dict ‘elem’ has no key named ‘mask’, and the added new key and value is occupying the memory and has not been freed even if the iteration is over. I am not sure if this is a problem of pytorch or python, or my coding logic. But thank god this problem could be solved by moving this part of code elsewhere