Hi @albanD. Without any breakpoints, the code still returns the same errors. To illustrate the issue further, I setup my code as follows:
for i, input_str in enumerate(MyDataLoader, 0):
        output = model(input_str)
        print(output)
        loss = sentence_loss(output)
        loss.backward()
        print('pytorch is fantastic!')


and set another breakpoint at print('pytorch is fantastic!'). On the first two examples, that breakpoint is hit. On the third example, the breakpoint at print('pytorch is fantastic!') is not hit and loss.backward() gets called twice, producing the error on the second call.
The structure of my code does follow what is above, but I am using the Transformers library to handle the finetuning and GPT2. I’ve simplified it in this post because it’s a bit more complicated. The issue might be related to Transformers so I will definitely check with them.
Is there anything I could print out for loss before and/or after the loss.backward() call that might be useful? I cal also see into it’s attributes when the breakpoints hit.