Following is my code, it’s same as you said, but it still has out of memory issue:
> for batch_idx, (inputs, targets) in enumerate(trainloader):
>         cur_iter = (epoch - 1) * len(trainloader) + batch_idx        
>         gpu_tracker.track()
>         if use_cuda:
>               inputs, targets = inputs.cuda(non_blocking=True), targets.cuda(non_blocking=True)  # GPU settings    
>         optimizer.zero_grad()
>         inputs, targets = Variable(inputs, requires_grad=True), Variable(targets)