NEW UPDATE:
I’ve finally found the source of leak. It does have something to do with the roi pooling module.
h = self.roipool(x, bh) The variable h here was still stored in the memory even though it was overwritten afterwards. And the behaviour repeats itself in every single step, which was causing the snowballing of memory leak.
Also, the variable bh here, which consists of a bunch of bounding boxes, was also not released after the forward pass, same for any subsequent steps. It is essentially these two variables, that were causing the memory leak.
For the variables o in o = self.roipool(x, bo) and bo in the “object stream”, same problem applies.
A possible reason for the not released memory is that these bounding boxes, including variable bh and bo, have variable length across different batches. I’ve heard that pytorch seems to have some problems with variable length batch data. But I’m not sure if this is indeed causing the memory leak.