I tried to set my variable to volatile and worked for me. I am thinking that maybe the first iteration the model allocate memory to some of variables in your model and does not release memory. At the second iteration , GPU run out of memory because the previous occupied variables.
Here’s the related link:
http://pytorch-zh.readthedocs.io/en/latest/notes/autograd.html?highlight=volatile 78
    ims = Variable(ims,volatile=true)
    masks = Variable(masks,volatile=true)

In this way, training set will not always in gpu memory.