Since you are emptying the cache it’s expected that the memory usage fluctuates, otherwise PyTorch would reuse the cached memory.
In case you are running out of memory after a while without seeing an increased memory usage in each iteration, I would guess that some inputs might be larger than others (could this be the case)?