Batch processing of sequences with different length
Hello, I have came up with something like this. q1 and q2 are just list of integer word indices. <SCODE> def SimilarityModule(nn.Module)
     def __init__():
         self.embedding = nn.Embedding(...)
     ....
    def forward(self, q1, q2):
        q1_embeds = self.embeddings(q1)
        q2_embeds = self.embeddings(q2)
        #mean over whole sequence
        q1_repre = torch.mean( q1_embeds, 0 ) 
        q2_repre = torch.mean( q2_embeds, 0 )

        dot_product = torch.dot( q1_repre, q2_repre ) / torch.norm( q1_repre, 2 ) / torch.norm( q1_repre, 2 )

        return dot_product
<ECODE> I have a question how properly create a batch of the sequences with variable length and feed it to the module? Thank you in advance for help!
octopusyo
smth
Thank you , Is there any way to work with “truly” variable length sequences? For instance, I would want to have a CNN with a RNN on top, I would also need masking?
octopusyo
You would use padding in the CNN, then take the output and pass it to pack_padded_sequence for the RNN.
jekbradbury
octopusyo
Masking is mandatory but the PyTorch RNNs natively support variable length sequences (created by pack_padded_sequence) and will correctly avoid processing the padding tokens, even for the reverse direction of a bidirectional RNN.
jekbradbury
Could you plz point to some example where variable length sequences are being passed in a batch
aradhyamathur
I do not understand what is really meant by “masking” - could anyone explain please?
josmi9966
flyaway