I’m back @apaszke . No del and gc.collect() did not help.
However reducing the number of workers from 8 to 2 did reduce the memory footprint by half, at the cost of increasing data loading time. Still, even when lowering the number of workers, the memory usage keeps growing (slower) from batch to batch, until it reaches a stable point. I guess you have implemented some kind of maximum caching for the queue, and the limit is satisfied. This is both a good point to see it constant and sad because it shows a problem I have no idea how to solve. Reducing the batch size should help also right ? as it is prepared on the CPU side, I am currently at 4096 with 3 inputs of 75x75, 51x51, 25x25.
He is my current train loop :
for i, (input, target) in enumerate(train_loader):
        # measure data loading time
        data_time.update(time.time() - end)

        target = target.cuda(async=True)
        input_25 = torch.autograd.Variable(input[0]).cuda()
        input_51 = torch.autograd.Variable(input[1]).cuda()
        input_75 = torch.autograd.Variable(input[2]).cuda()

        target_var = torch.autograd.Variable(target)

        # compute output
        output = model(patch25=input_25, patch51=input_51, patch75=input_75)
        loss = criterion(output, target_var)

        # debug loss value
        # print('raw loss is {loss.data[0]:.5f}\t'.format(loss=loss))

        # measure accuracy and record loss
        prec1, prec5 = accuracy(output.data, target, topk=(1, 5))
        losses.update(loss.data[0], input[0].size(0))
        top1.update(prec1[0], input[0].size(0))
        top5.update(prec5[0], input[0].size(0))

        # compute gradient and do SGD step
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        # measure elapsed time
        batch_time.update(time.time() - end)
        end = time.time()

        if i % args.print_freq == 0:
            print('Epoch: [{0}][{1}/{2}]\t'
                  'Time {batch_time.val:.3f} ({batch_time.avg:.3f})\t'
                  'Data {data_time.val:.3f} ({data_time.avg:.3f})\t'
                  'Loss {loss.val:.4f} ({loss.avg:.4f})\t'
                  'Prec@1 {top1.val:.3f} ({top1.avg:.3f})\t'
                  'Prec@5 {top5.val:.3f} ({top5.avg:.3f})'.format(
                   epoch, i, len(train_loader), batch_time=batch_time,
                   data_time=data_time, loss=losses, top1=top1, top5=top5))

        del input_25
        del input_51
        del input_75
        del input
        gc.collect()