Hi Richie!



 richieYT-wan:

I didn’t quite understand that the ground truth couldn’t be one-hot encoded, while the output of my network should have switched shape [batch, N_classes, length] instead of [batch, length, N_classes].


Just to be clear, if you want to use pytorch’s CrossEntropyLoss, you
have to do it this way.
CrossEntropyLoss is written to take as its target integer class labels
rather than one-hot-encoded labels and doesn’t have a built-in feature
to use one-hot encoding.  Similarly, CrossEntropyLoss expects the
second dimension of its input to be the class dimension, so that input
has shape [mini_batch, N_classes, d1, d2, ...], where d1, etc.,
are optional additional dimensions for the “K-dimensional case.”
(That’s just how it works.  If you need something else, you would have
to write your own version or write a wrapper for CrossEntropyLoss.)


If I slice my tensor and then run two different CrossEntropyLoss for each of the subtensor, will Autograd still work ?


Yes, doing this makes perfect sense.
I don’t understand your use case or your “positional encoding,” so let
me use an artificial example:
Let’s say you have a set of images and each image contains both a
picture of an animal – cat, dog, mouse – and a digit – 0 though 9.  You
want your network to classify both at the same time.
The input to your network (not to the loss criterion) is a batch of images
with shape [mini_batch, height, width], and your ground-truth
labels have shape [mini_batch, 2].  The “animal” labels are
labels[:, 0], a vector of length mini_batch, and are integer class
labels with values 0 (cat), 1 (dog), and 2 (mouse).  The digit labels
are labels[:, 1] (also, of course, of length mini_batch) and have
values 0 through 9.
So far, so good.
Your network architecture is whatever it is – perhaps some initial
convolutional layers because you’re classifying two-dimensional
images – followed by some fully-connected layers.  But your final
fully-connected layer should have nAnimal = 3 plus nDigit = 10
output features so that the output of your network (the input to your
loss criteria) is a tensor of shape [mini_batch, nAnimal + nDigit].
Now you slice input and labels and apply CrossEntropyLoss twice:
lossAnimal = torch.nn.CrossEntropyLoss() (input[:, :nAnimal], target[:, 0])
lossDigit = torch.nn.CrossEntropyLoss() (input[:, nAnimal:], target[:, 1])
lossTotal = lossAnimal + lossDigit
lossTotal.backward()

It’s fully legitimate to build your total loss out of multiple losses – in this
case two cross-entropy losses – and it will work just fine with autograd.
As you train your network by backpropagating totalLoss, your network,
and, in particular, your final fully-connected layer, will 'learn" that the first
nAnimal output features of the final layer are the predicted raw-score
logits for the animal in the image, and that the remaining nDigit output
features are the predicted logits for the digit in the image.
Best.
K. Frank