Thanks, I think I understand the logic but fail to understand what to do with the loss tensor before calling backward
In my forward function I have this now:
loss = F.cross_entropy(logits.view(-1, logits.size(-1)), targets.view(-1), reduction='none')
…here I perform my matrix mul on the loss tensor …
After that, should I do a simple loss = loss.mean() before calling loss.backward? I don’t understand how that will add a weigthed bias to the gradients? But if it works I will go for it, as simple as it sounds…
Or should I call loss[n].backward() for each batch sequence? Would that make any difference?
EDIT: Oh, nevermind. I figured that was an unneccessary question to ask when the answer was one test and therefore ~2 minutes away