In Translation with a Sequence to Sequence Network and Attention, the author trained seq2seq model with batch size = 1.
I want to train seq2seq model with batch size bigger than 1.
To deal with the different length of each input sequence, we can use PackedSequence as our input.
However, I found out that PackedSequence can only be used as the input of RNN modules.
There are some layers such as nn.Embedding in encoder and decoder, which do not take PackedSequence as input.
I think that maybe the Variable can be packed as PackedSequence before entering the rnn layer, and use torch.nn.utils.rnn.pad_packed_sequence to pad the output of rnn layer.
But here’s the problem. Due to the attention mechanism, the sequence needs to be processed word by word. It means that I need to divide the PackedSequence into smaller PackedSequence with size ( 1, batch_size, * ). It become less intuitive.
Anyone have any idea to solve this problem?
Thanks for answering !