nvidia-smi might show a higher usage, since PyTorch uses a caching memory allocator.
Have a look at the memory management 386 for more information.
Your current problem is that your GPU runs out of memory, if you don’t delete logits?
Could you post your code then please? It might be due to unwanted storing of logits, thus keeping the computation graph alive.