Hello John!



 John_J_Watson:

I am not sure what is the correct value to pass the loss function - raw logits or the values obtained by passing them to softmax().


It depends on what you are trying to do, and what your loss function is.
It sounds as if you might be working on a twenty-class classification
problem.  In such a case, you would typically use CrossEntropyLoss
as your loss function.  Your model would output directly the results of
its last fully-connected Linear layer as logits.  CrossEntropyLoss
would take these logits and a batch of integer class labels as its
inputs.  You would not pass anything through softmax() (although
a softmax() is, in effect, built into CrossEntropyLoss).


In the little tests I am doing, I do not actually see a difference and hence my question.


I’m not sure exactly what you are doing, what your tests are, and how
you are looking for a difference.
But here is a (pytorch version 0.3.0) sample script that shows the
effect of adding softmax():
import torch
torch.__version__
torch.manual_seed (2020)

some_numbers = torch.autograd.Variable (torch.randn (5), requires_grad = True)
some_numbers

some_outputs = some_numbers
loss = some_outputs.sum()
loss
loss.backward()
some_numbers.grad

some_numbers.grad.zero_()
some_outputs = some_numbers
some_outputs = torch.nn.functional.softmax (some_outputs, 0)
loss = some_outputs.sum()
loss
loss.backward()
some_numbers.grad

And here is the output:
>>> import torch
>>> torch.__version__
'0.3.0b0+591e73e'
>>> torch.manual_seed (2020)
<torch._C.Generator object at 0x000001BAF61C6630>
>>>
>>> some_numbers = torch.autograd.Variable (torch.randn (5), requires_grad = True)
>>> some_numbers
Variable containing:
 1.2372
-0.9604
 1.5415
-0.4079
 0.8806
[torch.FloatTensor of size 5]

>>>
>>> some_outputs = some_numbers
>>> loss = some_outputs.sum()
>>> loss
Variable containing:
 2.2911
[torch.FloatTensor of size 1]

>>> loss.backward()
>>> some_numbers.grad
Variable containing:
 1
 1
 1
 1
 1
[torch.FloatTensor of size 5]

>>>
>>> some_numbers.grad.zero_()
Variable containing:
 0
 0
 0
 0
 0
[torch.FloatTensor of size 5]

>>> some_outputs = some_numbers
>>> some_outputs = torch.nn.functional.softmax (some_outputs, 0)
>>> loss = some_outputs.sum()
>>> loss
Variable containing:
 1
[torch.FloatTensor of size 1]

>>> loss.backward()
>>> some_numbers.grad
Variable containing:
 0
 0
 0
 0
 0
[torch.FloatTensor of size 5]

As you can see adding the line
some_outputs = torch.nn.functional.softmax (some_outputs, 0)

causes the loss to change to 1 (from 2.2911), and the gradient to
change to [0, 0, 0, 0, 0] (from [1, 1, 1, 1, 1]).
Typically you would call loss.backward() to calculate the gradient
of your loss function with respect to your model parameters.  Then you
would call optim.step() to adjust your model parameters a little bit in
the direction opposite to your gradient.  So a different gradient means
different updated model parameters, and this means different results.
Best.
K. Frank