If your domain vocabulary is totally different, then you probably do not want to adapt an existing, pretrained LM model to your domain-specific corpus.  In this case, it probably makes more sense to train an LM from scratch.
You can check how different your domain is from an existing pretrained vocabulary by running the corresponding pretrained tokenizer against a sample of your domain’s text and counting the number of unknown tokens that show up.
In my experience, the pretrained tokenizers like BPE will not return any unknown tokens if your domain-specific corpus is in the same language (e.g., English).