When I did machine translation task(using the Transformer model), I find a question:
As I want the model to predict the  token but not have it be an input into the model, I simply slice the <eos> token off the end of the sequence. Thus:
trg = [sos, x_1, x_2, x_3, eos]
trg[:-1] = [sos, x_1, x_2, x_3]
But when I use torchtext generate my datasets, I find the last elements of sentence is the <pad> token, such as:
trg = [sos, x_1, x_2, x_3, eos, pad, pad, pad]
trg[:-1] = [sos, x_1, x_2, x_3, eos, pad, pad]
so i can’t slice the <eos> token, how can I solve this question?