Wondering if I have a similar issue. GPU memory usage is steady for some epochs and it spikes up and eventually encounters CUDA out of memory error.  Is it possible for GPU memory to increase like that after so long into epochs?
Using pytorch-lightning 0.8.5 with pytorch 1.6.0
image405×669 21.9 KB