I made the changes as you suggested. Now, I have only one learnable layer in my network.
This is what my network looks like -
scores = torch.matmul(self.linear(mention_representation), label_representation.t())

where self.linear = nn.Linear(300, 300, bias=False), mention_representation and label_representation are pretrained embeddings.
Observations:
The loss decreases to a certain point and then starts increasing, but the rate at which the loss increases is very low.
Thoughts/questions:
Do you think torch.matmul causes the issue?