I’m a bit confused regarding the initial issue.
Are you seeing an icreased memory usage in each epoch or just a varying one?
The latter case might be expected, especially if you are manually clearing the cache (which will not save any memory, but slow down your code).
Is the allocated memory higher using apex or just the reserved?
You can check it via torch.cuda.memory_allocated() and torch.cuda.memory_reserved().
The reserved memory might be higher, if some intermediate tensors are additionally created and it also depends on the opt_level you are using.
That being said, we recommend to check out the native mixed-precision 1 implementation, which can be used by installing the nightly binaries or by building from source.