THE PROBLEM IS SOLVED
The memory leak was caused by a python wrapper of the roi pooling function 93
The attribute self.output contains the output of the roi pooling module. But since the output will be returned anyway, there really isn’t any point saving it as a private attribute of module. And as a consequence, the memory taken by self.output was somehow never released and stacked up over training steps.
After commenting out this attribute, the memory leak was gone.