Thank you for your answer. I used torch.cuda.empty_cache() followed by torch.cuda.memory_summary() to monitor the memory. I noticed an increasing gap between the total allocated memory and the total freed memory. This gap gets larger with the number of iteration. This is why I assume an issue like some memory created at every iteration not being freed.