Could you explain what “locked in GPU memory” means and how you are measuring it?
Note that PyTorch caches the GPU memory, so after deleting a tensor, the memory will not be immediately released to avoid synchronizing (and thus expensive) memory reallocations.