Hey @ptrblck thanks for replying!
So my use case involves predicting relationships between words in text, so my input is a dict where the key is a 4-tuple (word, pos_tag, dep_tag, dir) and the value is the frequency of the corresponding path. I feed these input into an LSTM, which uses the tuple and the count to predict the type of relation.
Regarding the 4-tuple, all of these are fed as indices, and inside the network, an nn.Embedding layer is used for each. Now since I want pos_tag, dep_tag and dir embeddings to update as the model trains, I let nn.Embedding layers for these terms to remain as they are. But for word embeddings, I precompute the embeddings using pretrained Wiki2Vec model 2. The embedding dimension is 300. Earlier I was using Glove word embeddings, but since they weren’t very well suited to my task, I decided to switch to Wiki2Vec.
Hope that explains my use case enough.
Also, could you please explain what you mean by inputs not being “well represented”?