Sure, you can concatenate vectors. For example, say you have


embed with embed.shape = (batch_size, seq_len, embed_dim)


custom with custom.shape = (batch_size, seq_len, custom_dim)


You can do:
X = torch.cat([embed, custom], 2)
Then X.shape = (batch_size, seq_len, embed_dim+custom_dim)