I make a try with Pytorch’s own memory helper function and find another weird phenomenon.
I add following code in every iteration
print('current memory allocated: {}'.format(torch.cuda.memory_allocated() / 1024 ** 2))
print('max memory allocated: {}'.format(torch.cuda.max_memory_allocated() / 1024 ** 2))
print('cached memory: {}'.format(torch.cuda.memory_cached() / 1024 ** 2))

And this is what I get
current memory allocated: 145.43115234375
max memory allocated: 2584.5517578125
cached memory: 7989.5

Looks like current allocated memory size is pretty normal but the size of max memory allocated and cached memory is abnormally high. I try to call torch.cuda.empty_cache() after each iteration, which does decrease the VRAM usage significantly but I still get OOM issue after running for a longer time.
So why does cached memory allocator eats up so much VRAM?