During the calculation of proposal and target iou in detectron2 framework, if OOM appeared, it would try to transfer it to CPU for calculation, but the error of gpu out of memory still occurred when moving to CPU
Traceback (most recent call last):
  File "main.py", line 76, in <module>
    loss_dict = model(data)
  File "/root/anaconda3/envs/python367/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/detectron2/detectron2/modeling/meta_arch/rcnn.py", line 124, in forward
    proposals, proposal_losses = self.proposal_generator(images, features, gt_instances)
  File "/root/anaconda3/envs/python367/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/yaogan/custom/kaggle_rpn.py", line 114, in forward
    losses = outputs.losses()
  File "/root/detectron2/detectron2/modeling/proposal_generator/rpn_outputs.py", line 333, in losses
    gt_objectness_logits, gt_anchor_deltas = self._get_ground_truth()
  File "/root/yaogan/custom/kaggle_rpn.py", line 57, in _get_ground_truth
    matched_idxs, gt_objectness_logits_i = retry_if_cuda_oom(self.anchor_matcher)(match_quality_matrix)
  File "/root/detectron2/detectron2/utils/memory.py", line 84, in wrapped
    return func(*new_args, **new_kwargs)
  File "/root/detectron2/detectron2/utils/memory.py", line 82, in <genexpr>
    new_args = (maybe_to_cpu(x) for x in args)
  File "/root/detectron2/detectron2/utils/memory.py", line 65, in maybe_to_cpu
    return x.to(device="cpu")
RuntimeError: CUDA out of memory. Tried to allocate 2.27 GiB (GPU 0; 10.76 GiB total capacity; 7.76 GiB already allocated; 334.69 MiB free; 9.60 GiB reserved in total by PyTorch)


Why ?