Hi Thomas,
Thanks for your reply. I guess my question is not clear enough. Here is a more concrete example:
Say I have two sentences of different length. When computing the embedding one the two sentence, one naive way to do this is simply using nn.EmbeddingBag with mode='sum' or mode='mean'. However, what I want is sort of attention on the embeddings, i.e. rather than w_1 + w_2 + … + w_n, I’m looking for a_1 * w_1 + … + a_n * w_n, where both the attention weight a and the embeddings w are learnable. So what I did is first use nn.Embedding to extract the vectors, then weight it. Finally, since multiple sentences may have different length, I used nn.EmbeddingBag to sum the corresponding word embeddings by providing the offsets.
The solution that you suggest seems to me like a dense version, where there will be lots of 0 in the matrix. I believe it will consumes way more memory. What I’m looking for is a memory efficient (sparse) way to deal with this. Please correct me if I’m misunderstand something