I don’t see any non-linearity being used here (eg: sigmoid or ReLU). Are your loss values decreasing during training? If not, your model is not learning.
Instead of using cosine_sim, I would rather do this:
a. Append the question-document pairs and pass them through the pretrained BERT model.
b. Add a linear layer for the output from the first token (assuming it’s CLS token) and use BCELoss.