Solved by ptrblck in post #14 
              
              
                You could try to see the memory usage with the script posted in this thread. 
Do you still run out of memory for batch_size=1 or are you currently testing batch_size=4? 
Could you temporarily switch to an optimizer without tracking stats, e.g. optim.SGD?