This is because of the caching allocator. https://pytorch.org/docs/master/notes/cuda.html#memory-management 164. Not sure what is the equivalent in c++, but the python way is to torch.cuda.empty_cache().