Here, I got a different problem of GPU memory overflow, I got to see that GPU memory requirement is incrementally increasing with the number of iterations and after 11k iterations, its all occupied and so is the error, while I think it should not increase with the iterations and the code which creates the problem.
Can you help me in identifying which variable’s size is iteration dependent here?
for i, (input1, target) in tqdm(enumerate(train_loader)):
    if args.gpu is not None:
        input1 = input1.cuda()#args.gpu, non_blocking=True)
    target = target.cuda()#args.gpu, non_blocking=True)
    output = model(Variable(input1))
    loss = criterion(output, Variable(target))
    prec1, prec5 = accuracy((output.data).cpu(), target.cpu(), topk=(1, 5))
    losses.update(loss, input1.size(0))
    top1.update(prec1[0], input1.size(0))
    top5.update(prec5[0], input1.size(0))
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    batch_time.update(time.time() - end)
    end = time.time()
    if i % args.print_freq == 0:
        batch_time=batch_time
        top1=top1
        top5=top5
        loss=losses