I did some experiments and found that the memory leak was in the code below:
for elem in target:
    mask = coco.annToMask(elem)
    mask = cv2.resize(mask, (self.w, self.h),
                              interpolation=cv2.INTER_AREA)
    elem['mask'] = mask
    elem['category_id'] = COCO_DICT[str(elem['category_id'])]
    elem['bbox'][0] *= w_ratio 
    elem['bbox'][2] *= w_ratio 
    elem['bbox'][1] *= h_ratio 
    elem['bbox'][3] *= h_ratio