Do you see the increased memory usage still in the first epoch, after the validation of the first epoch or at the beginning of the second epoch?
I cannot see anything strange in the code, so if almost all your memory is used already, this might be due to memory fragmentation. Could you try to remove the instantiation of the optimizer inside run_training, just initialize it once, and pass to run_training?