Thanks a tonne for this Frank - it really helps my understanding of things.



 KFrank:

(It’s not clear to me what you mean by “train.” If you pass outputs
to a loss function, call loss.backward() , and then take an optimizer
step, you will get different results if you leave out the softmax() .)


Could you please elaborate on this? Infact, this is what I am doing, and I am not sure what is the correct value to pass the loss function - raw logits or the values obtained by passing them to softmax(). What should be the expected difference? In the little tests I am doing, I do not actually see a difference and hence my question.