Solved by vdw in post #2 
              
              
                Sure, you can concatenate vectors. For example, say you have 


embed with embed.shape = (batch_size, seq_len, embed_dim)


custom with custom.shape = (batch_size, seq_len, custom_dim)


You can do: 
X = torch.cat([embed, custom], 2) 
Then X.shape = (batch_size, seq_len, embed_dim+custom_dim)