hello again,
I’ve had a look at the BioBERT article.   Fig 1 does look as though they (pre-) train with medical texts using BERT base weights, but Section 3.3 clarifies that actually they used the standard BERT base vocabulary and that the training of BioBERT effectively started with the standard Wiki+Books data and then was followed by the medical texts.   So, the BioBERT article is all about fine-tuning a pre-trained BERT model.
There can be as many training steps as you care to imagine when doing NLP with BERT.   So far as I know, people only use two terms for it: pre-training and fine-tuning.   We could really do with a term for the intermediate steps.   I call it intermediate-training.
Pre-training:   that which somebody else has already done.
Intermediate training: further modification, for example using language modelling with domain-specific texts.
Fine-tuning: training to the task of interest.
The Intermediate stages can also include other steps such as Distilling or Pruning, and might include Freezing some or all of the layers.
Note that the BioBERT article says “pre-trained on biomedical corpora”, but I would classify that as Intermediate training.   I suppose, if you pick up their finished model, you would then consider both the Wiki+Books and the medical corpus as pre-training.
If anyone knows a standard classification of what counts as “pre-training” I’d like to know.  @BramVanroy maybe?