@lewtun @BramVanroy - thank you! Here is what I have gathered from your responses:

We can aggregate sub-word embeddings to obtain word embeddings, but the performance impact needs to be tested on the down-stream task.
Context insensitive embeddings from BERT etc will perform worse than word2vec, glove, etc. I remember hearing this point in Nils Reimers’ video on sentence transformer.

But I am still not clear about two things:

Isn’t it a common task to obtain word embeddings from a fine-tuned LM and then use them for a specific task - like what ElisonSherton has described? Can you please point me to better approaches apart from just aggregating the sub-word embeddings?
Also, some code snippet showing how to extract the embeddings from an LM will be helpful.

Sincerely,
Vishal