If the OOM happens on the second iteration then it may be because the variables inside the loop are still alive during the second run. Python has function scoping (not block or loop scoping) so any variables declared during the first iteration remain alive in subsequent iterations until they are redefined.
Here is another post about this:

  
    
    
    Increase the CUDA memory twice then stop increasing 
  
  
    I have the code below and I don’t understand why the memory increase twice then stops 
I searched the forum and can not find answer 
env: PyTorch 0.4.1, Ubuntu16.04, Python 2.7, CUDA 8.0/9.0 
from torchvision.models import vgg16
import torch
import pdb

net = vgg16().cuda()
data1 = torch.rand(16,3,224,224).cuda()

for i in range(10):
    pdb.set_trace()
    out1 = net(data1)


first stop, this is what data1 and vgg16  take 
[1]

second stop, this is what the intermediate status of vgg16 take 
[2…
  


See my comment about how you can rewrite your program to avoid two versions alive at once.