I have a sequence model and I want to concatenate a same piece of small embedding to the input. Specifically, that looks like
if the original embedding is ABCDEFG, the expected one would be
A  B  C  D  E  F  G
X  X  X  X  X  X  X
From a coding aspect, that’s something like:
embed_0_out = embed_0(input_0) # sequence input
embed_1_out = embed_1(input_1) # single input
        
embed_combined_out = torch.cat([embed_0_out, embed_1_out.unsqueeze(1).expand(batch_size, input_0_len, 10)], 2))

However, it seems like the weight for embed_1 is not changing after each iteration, suggesting that it is not trainable. Thus, is there any way to make it trainable?