HI,
You can check the doc about how we manage the CUDA memory here 38.
In particular, this will explain why the memory is not returned to the OS when you delete your model.
For trying batch sizes, there are many things that can change the way the memory is allocated on the GPU and so, because of the caching allocator, will slightly change the memory usage. This is unfortunately expected and so you want to keep a liitle bit of extra memory free to make sure you don’t have any issue.

everything should fit into memory, although it doesn’t, it crashes consuming the entire gpu memory

I don’t think you took into account the memory used for the gradients for each parameter. And the memory used by the different states needed to compute the backward pass.
These states are expected to be the most consuming in terms of memory. If you have big memory limitation, you can use modules like the checkpoint one here 19 to reduce the memory used by the states at the cost of doing extra computations.