My problem is more likely to sequential decoding. Each target can be repeated, like [5,2,1,0,1]. I padded the targets and, used nn.CrossEntropyLoss() with ignore_index=-100 parameter. But still no improvement…
Thanks for your answer.