Hi
I get ‘RuntimeError: CUDA out of memory’ in train mode. I can not reproduce the problem anymore,Tried all of these:
. gc.collect()
.torch.cuda.empty_cache()
. set the batch size equal to 1
What is strange, is that the EXACT same code ran fine before. When I tried to run the same code with slightly different hyperparams (doesn’t affect the model,like learning rate and decay) it breaks during the first iteration of the first epoch. Even when I try to run the same hyperparams as my first experiment, it failed. Can you hepl me?thanks.