To the thread - medical notes related tasks are rather niche because it involves a lot of domain expertise (meaning labeling is time-consuming and super expensive) and there’s really not that much data to train on compared to other domains.
I’ve been wondering about the same situation myself (medical domain with clinical notes or needing to do literature search). I found that sentence vectors (using sentence transformers) have some limitations. It doesn’t work all that well with the pre-trained models and I found that the similarity scores more or less act as fancy regex functions, and doesn’t do so much to capture semantics.
You saw this as well in the Cord-19 Kaggle challenge where everyone implemented roughly the same idea but none of the results were that convincing.
I think there’s a couple of things you could potentially explore:

A pretrained paraphrase task may be better than similarity task
Going back to simple vectors (like fastText) and doing your search query on those embedded terms (but this really will only benefit literature search rather than notes due to corpus size)
Knowledge graph creation with embeddings.
Simple training exercise