@apaszke, @albanD, I also tried to achieve this. As you have said, doing backward() for each sample is slow compared to accumulating the loss, doing one average and then doing backward.  Here is my code
num_epoch = 10
real_batchsize = 100 # I want to update weight every `real_batchsize`
for epoch in range(num_epoch):
    total_loss = Variable(torch.zeros(1).cuda(), requires_grad=True)
    for batch_idx, (data, target) in enumerate(train_loader):

        data, target = Variable(data.cuda()), Variable(target.cuda())
        output = net(data)

        total_loss = total_loss + loss

        if batch_idx % real_batchsize == 0:

            ave_loss = total_loss/real_batchsize
            ave_loss.backward()
            optimizer.step()
            
            total_loss.data.zero_()
            optimizer.zero_grad()


The above code will produce an error:

RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

I have looked up this issue, but I am not very clear. Just feels that we need a new total_loss after each weight update, so I replace the line
 total_loss.data.zero_()

by
total_loss = Variable(torch.zeros(1).cuda(), requires_grad=True)

Now it seems to work. But I am 100% sure if I have done it right. Can you give any advice on how to do it properly?