The caching allocator takes memory in blocks and if the block isn’t completely empty, it cannot be deallocated.
I can recommend torch.cuda.memory_summary() for looking at it in more detail.
Best regards
Thomas