Is this the RAM or the GPU memory?
Skimming through the code I cannot find anything obviously wrong.
Could you map the seconds to epochs or iterations? What happens after 300 seconds?
Are you starting the validate the model or does a new epoch start?