How did you increase the batch size by removing the cache?
Note that the memory stored in the cache can be reused.
If you empty the cache, PyTorch would have to reallocate the memory.
Freeing the cache should not save memory, but just slow down your code due to free and alloc calls.