Hi all, same thing is happening to me, but not always, and that’s the weird part, the inconsistency. I’m using CPU only on gcloud VM. Note that I had a successful run of 100
epochs. I changed learning rate and input/hidden dimensions very slightly, and the memory increase issue happened until it killed my job.