Understanding pre-training and fine-tuning
I am confused about what these two steps actually do to the model. I would have assumed that pre-training is unsupervised (i.e. no labels) and, thus, the only thing that can be 'learned' is the embedding representations of all tokens. You can then use this pre-trained model (which is an 'empty' model but with pretrained vector representations of tokens) to actually train the whole mode.

However, from the things that I read that is not the case. It's not only the embeddings that are pretrained, but also the whole model. Is that right? How can that be, without labels?

For context: I want to train a Bert model on a new language and I am trying to figure out which steps I have to do to get there. Can I load custom embeddings, and do I then still need to pretrain the rest of the model (but how, which labels?).

I know that my question is quite general, but the difference between pre-training and fine-tuning is not clear to me. Pre-training should be 'done once per language', but then what does the model actually learn? It it simply a seq-to-seq pre-training?
Maybe a good introduction on the topic are the writings of @sebastianruder:
- http://ruder.io/transfer-learning/
- https://thegradient.pub/nlp-imagenet/
- the ULMFiT paper: http://nlp.fast.ai/classification/2018/05/15/introducting-ulmfit.html

Regarding your specific question of training a Bert model on a new language, the computing requirements for training BERT are very high. You would probably be better with a version of ULMFiT. There is a huge multilingual initiative on the fast.ai forum that you can check out. One entry point is [here](https://forums.fast.ai/t/language-model-zoo-gorilla/14623).<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
