[CamemBert] About SentencePiece training
## ❓ Questions & Help

According to the paper, SentencePiece uses a vocabulary of size of 32k subword tokens, learned on 107 sentences sampled from the pretraining dataset. How the sampling was performed? The chosen size of the vocabulary (32K subwords token) is related to the pretraining dataset in some way? Or it is an arbitrary choice?

Thank you.
ping author @louismartin :)<|||||>Hi @loretoparisi, 
We sampled 10**7 lines randomly from the pretraining corpus.
The size of the vocabulary was chosen to somewhat match the original BERT paper which used a 30k wordpiece vocabulary, so yes it's mostly arbitrary. <|||||>@louismartin thanks a lot for the details. I was wondering about the 32k size if this could have been biased by the language...w<|||||>Yes maybe there is a more adapted vocabulary size, we did not investigate that :)

Can I close the issue now?<|||||>Adding a reference to https://github.com/google/sentencepiece/issues/415<|||||>@louismartin: 
> We sampled 10**7 lines randomly from the pretraining corpus.

May I ask how did you come up with that number? I'm trying to figure out how many lines I should select to train a model. Assuming I have access to 1 billion rows of ngrams with mean length of 7 words; I'm not sure how many random lines/ngrams would be enough to train a tokenizer with fixed vocab of size 50k?<|||||>I think this is more of a resource allocation question. How much time or compute do you want to allocate to training your tokenizer? Alternate phrasing: why wouldn't you train your tokenizer on the full corpus?<|||||>@julien-c Well sure, that's a valid point. In theory I can train a tokenizer on full corpus by setting the fixed size for the vocabulary. It will just take more and more time (& possibly more compute resources) with increased size of dataset. I was wondering if there's any correlation between quality of fixed threshold vocabulary generated from increasing size of training dataset. I can see that this may be task dependent and requires iterative experiments. Is there any paper that I can look into regarding this? Thanks! <|||||>PS: did you check out [`tokenizers`](https://github.com/huggingface/tokenizers)? It is pretty fast 😄 

I've trained a byte-level BPE on 10 GB of text in ~15 minutes.<|||||>@julien-c are you suggesting that, thanks to 🤗 amazing library `tokenizer` we could potentially train the sentence piece tokenizer without setting up a boundary?
This means that, potentially, current models could improve a lot: from 32K subwords tokens to let's say 1M, what will happen?<|||||>Those are two different things: size of vocab, and size of corpus that you train your tokenizer on.<|||||>@julien-c that's true, and it also seems to have no clear relation in terms of final overall accuracy. Let's say we take as metrics the PPL, and we consider a fixed corpus size and vary the vocab size in batches of 8K, like: 8K, 16K, 32K, 64K, 128K, until we are closer to the whole size of the non unique tokens vocabulary. What will be the related PPL for each training?

(PPL or BLEU, or other...)