I did a experiment in resnet50_maskrcnn, here is the code:
print(torch.cuda.memory_reserved())
print(torch.cuda.memory_allocated())
x = self.extract_feat(img)
print(torch.cuda.memory_reserved())
print(torch.cuda.memory_allocated())
model had call eval(), so the is_training is False
201326592(about 192MB) ---- print(torch.cuda.memory_reserved())
193334272(about 184.377MB) ---- print(torch.cuda.memory_allocated())
before the inference pytorch reserved about 192MB, I can understand that.
x = self.extract_feat(img)
this is resnet backbone, x consume about 87306240(83.262MB)
1304428544(1244MB) ---- print(torch.cuda.memory_reserved())
1267727360(1208.999MB) ---- print(torch.cuda.memory_allocated())
My question is why after the backbone inference pytorch hold so many GPU memory? There is only x in the GPU and it only consume about 83.262MB, what about 1GB(1244 - 83.262) GPU memory used for?