Hello Ege!



 ekarais:

In the second iteration of the training loop, the following runtime error is produced:
RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

        logits = model(x)
        
        #compute loss
        for i in range(18):
            loss_array[i] = criterion(logits[:,i*11:(i+1)*11], torch.argmax(labels[:,i*11:(i+1)*11], dim=1))

        for i in range(9):
            loss_array[18+i] = criterion(logits[:,198+i*8:198+(i+1)*8], torch.argmax(labels[:,198+i*8:198+(i+1)*8], dim=1))
        
        for i in range(6):
            loss_array[27+i] = criterion(logits[:,270+i*11:270+(i+1)*11], torch.argmax(labels[:,270+i*11:270+(i+1)*11], dim=1))

        loss = torch.sum(loss_array)
        
        # update the weights
        optimizer.step()



I’m not entirely sure what is going on here.  You don’t say what
loss_array is, but since you call torch.sum (loss_array),
I will assume that loss_array is some kind of pytorch tensor.
If so, indexing into loss_array multiple times could be your
problem.
(You also don’t say what criterion is.  Let me assume that is is
criterion = torch.nn.CrossEntropyLoss(), and therefore that
criterion (...) returns a pytorch tensor of shape [1], that is
a single number packaged as a tensor.)
Try something like:
        #compute loss

        loss = 0.0   # python scalar

        for i in range(18):
            # loss will become a pytorch tensor
            loss = loss + criterion(logits[:,i*11:(i+1)*11], torch.argmax(labels[:,i*11:(i+1)*11], dim=1))

        for i in range(18):
             # etc., ...

        loss.backward()




When I specify retain_graph=True when calling backward the first time, and set it to False in the subsequent iterations, I then receive the following error message:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [200, 336]]...

        # backward pass
        loss.backward(retain_graph=first)
        first = False
        train_loss += loss.item()



If my theory is right, this use of retain_graph = True is incorrect,
and, rather than fixing the real issue, is just hiding it.  So just call
loss.backward(), as outlined above, without specifying retain_graph.
Good luck.
K. Frank