The memory usage is most likely not showing the real used memory, but the cached memory by PyTorch.
Have a look at the Wikipedia article on Fragmentation 2. The table in " External fragmentation" gives a good idea of my assumption. While your old model might have taken the B block of memory, no new model bigger than B can fit in this gap, which will thus create a new block after C.