I am pre-training on product reviews written in Argentinean Spanish to generate meaningful embeddings that I can later use to understand properties about a review such as: what aspects of the purchase is it referring to, how positive or negative was the buying experience etc.
There is a great deal of data 1 about this and the domain-language is quite specific. Using a pretrained multilingual model has two problems:

The vocabulary is generally very large and most of the tokens are irrelevant for my use case (this makes training and inference slower).
The vocabulary misses some domain-specific language idioms that are frequently used by users and are important to understand the overall sense of the phrase.

This is why I think pretraining would be a better choice for my use case.