Hey
I’m trying to compute the unigram (bag of words) log-probability of a batch of sentences and I’m having trouble with the matching between predictions and targets.
To exemplify the problem:
For each sentence, in the batch, I compute a representation of it, which is then used to estimate the probability of each word in the vocabulary being in the corresponding sentence. My targets consist of the ids of the tokens in the sentence, padded to the max sentence length in the batch. E.g.:
sentences:
[[‘The’, ‘sky’, ‘is’, ‘blue’, ‘.’],
[‘The’, ‘tail’, ‘of’, ‘the’, ‘dog’, ‘.’]]
-1 for padding
targets:
[[0, 1, 2, 3, 4, -1],
[0, 5, 6, 0, 7, 4]]
preds:
[[0.2, 0.1, 0.15, 0.05, 0.3, 0.05, 0.05, 0.1],
[0.15, 0.25, 0.15, 0.05, 0.05, 0.1, 0.05, 0.2]]
So:
I could always implement my own loss function, using the ‘targets’ as indices of the elements of ‘preds’ that are to be selected (ignoring the pad index), take the log and sum of those (and average over batch). (I’m not actually asking any question in this point. Just as a check in case someone has a better solution )
But maybe Pytorch already has something like this implemented, some variation of the cross entropy function, but, if so, I haven’t been able to find it.
So, does it exist?