For Question Answering tasks, can we use nn.Embedding to represent role embeddings? In other words, the agent role embedding and the user role embedding are both trainable. The reasoning behind this is to use some embedding representations for agent and user utterances (GloVe, fastText, nn.Embedding, BERT embeddings, etc.), and we could add these trainable embeddings to the utterance representation according to the role utterance to help the model distinguish agent and user utterances.
For example, the authors of Multi-domain Dialogue State Tracking as Dynamic Knowledge Graph Enhanced Question Answering 3 apply this idea (search for role embedding on the .pdf). I would like to do something similar with torch.