CUDA out of memory error
I ran my net with a large minibatch on the GPU with problem, and then ctrl-c out of it. However when I try to re-run my script I get: RuntimeError: cuda runtime error (2) : out of memory at /home/soumith/local/builder/wheel/pytorch-src/torch/lib/THC/generic/THCStorage.cu:66 Nothing changed and it just worked before. Is the memory on the GPU not being released? Is there a way to force the GPU memory to release its memory before a run? Thanks.
Kalamaya
apaszke
nvidia-smi shows me using 8GB out of the 12 GB… What puzzles me is that literally just stopped with a large minibatch size that training just fine, but then when I re-start the python script it craps out.
Kalamaya
apaszke
Kalamaya
Kalamaya
Hi, Can you give me some suggestions? Thank you so much.
mjchen611
And when I set num_workers = 0,the (RAM, but not GPU) memory not increases largely with the increase of epoch… Can you give me some suggestions or instructions about the problem? Thank you so much.
mjchen611
i face the same error…is there a way to fin the variables which are getting cached on gpu
Harikrishna.Vydana