vgoklani:

what sort of performance metrics should one expect to see when pre-training?


Typically GLUE / SuperGLUE in English, or finetuning on your target domain should be also ok for a model targeted for that domain I think.



 vgoklani:

how long should we pre-train


 Scaling Laws for Neural Language Models 25 should help. In short, it depends on the size of model and how much cost you can afford or how much performance you want. I remembered huggingface has a calculator for this.