Thank you for a reply! Yes, that was set_trace with the following trace function.

  
      gist.github.com
  
  
    https://gist.github.com/MInner/8968b3b120c95d3f50b8a22a74bf66bc 1.2k
gpu_profile.py
import datetime
import linecache
import os

import pynvml3
import torch

print_tensor_sizes = True
last_tensor_sizes = set()
gpu_profile_fn = f'{datetime.datetime.now():%d-%b-%y-%H:%M:%S}-gpu_mem_prof.txt'
This file has been truncated. show original




  
  
    
    
  
  


I must have figured out the source of the leak by the way. It was due to the fact that significant portion of the code like variable allocation and intermediate computations was located within a single python function scope, so I suspect that those intermediate variable were not marked as free even though they were not used anywhere further. Putting a lot of del's kind of helped, but just isolating each individual step of computation into a separate function call so that all intermediate variable are automatically freed in the end of scope seems to be a better solution. Does that sound reasonable in context of pytorch?
I wonder if there could be some sort of “semantic garbage collection” that could detect that variable is not used anywhere further, thus could be freed.
Thanks!