Hi Jung, thanks for the reply.
If I had to pick a domain name, I would say medical notes. It contains a lot of technical terms but it is written quickly with abbreviations and acronyms galore.  I am planning on using BioBert, SciBert, or ClinicalBert as the pretrained base model.
I think the issue I have is that there isn’t an NLI dataset I can use for my niche domain. If I understand correctly, any data source can be used for MLM and NSP, but I don’t have the means of creating a labeled textual entailment (NLI) dataset, and I’ve heard that this type of dataset is the best for creating good sentence vectors.
See this paper 6 and this article 9 for reference about sentence embeddings.