ptrblck:

Is your model with the Dropout working or are you getting the OOM error now all the time?


OOM when I add dropout layers. Without dropout, I could fit [70 6000 6000 4] model on 1 GPU. (Any model with dropout layer is running into OOM)



 ptrblck:

Maybe a process still lives and uses all your GPU memory? Could you check it? I don’t know, if nvidia-smi works on a windows machine.


I use os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID" os.environ["CUDA_VISIBLE_DEVICES"]="0,1" to run it on specific GPUs. And no, these GPUs use 0 memory when experiments are not run. Checked via nvidia-smi



 ptrblck:

Did you observe the memory? Is it growing once you add Dropout to your model?


Yes, memory keeps growing when I try training models with dropout layer. I’ve trained bigger models (more parameters) on 1 12Gb GPU. But the model as small as one mentioned above, runs into OOM.



 ptrblck:

The volatile flag is deprecated. Since you are using pytorch 0.4.0 , you should use with torch.no_grad() instead.


Noted