Unigram frequencies in GPT-2 or XLnet?
Question: does the GPT2 or XLnet tokenizer contain unigram frequencies? From the discussion here (https://github.com/huggingface/pytorch-transformers/issues/477), it looks like the tokenizer from TransformerXL has it, but I'm not sure if the same applies for GPT2 and XLnet. If they do contain unigram frequencies, can you point me to the objects in the GPT2/XLnet's tokenizer that have this frequency information?
XLnet tokenizer utilizes SentencePiece and you can use it's score for unigram as relative frequency with something like `math.exp(XLNetTokenizer.sp_model.GetScore(token_id))`.<|||||>Ah perfect. So the raw score gives the unigram log probability, and the exp of it gives the normalised frequency.

How about GPT-2? Is this information saved in the tokenizer?<|||||>As far as I see, GPT-2 tokenizer does not contain frequency information. What's more tokens in dictionary are not ordered according to frequency (so it is not possible to estimate frequency assuming Zipf's distribution), but according to length.<|||||>Many thanks for looking into this. I suppose I have no choice but to mine unigram frequencies myself... (at this point I am looking to use a similar web corpus, such as https://skylion007.github.io/OpenWebTextCorpus/ to do this; if you have any pointers please chime in).<|||||>Hello @jhlau
I was looking for a similar unigram frequency for GPT-2. Would you happen to have acquired (or created) such a list and be willing to share?<|||||>There you go: https://drive.google.com/file/d/1FhObTkvhT46Xy-Vyqi-gku8Ho2c1XvHx/view?usp=sharing

Python3 pickle file. Unigram mined based on the openwebtextcorpus linked above.<|||||>Hi @jhlau 
Can you please explain how you mined the unigrams?
I looked at your file (thank you for sharing!) and it seems to me as if the keys are case-sensitive and that there was some method like SentencePieces or BPE used (I noticed keys like "ing" while very common verbs are not included). I'd like to be able to tell the frequency of a word, how should I go about it?
Thank you!<|||||>GPT-2 uses BPE, so the openwebtextcorpus is tokenised with BPE, and then unigram frequencies are collected.<|||||>> There you go: https://drive.google.com/file/d/1FhObTkvhT46Xy-Vyqi-gku8Ho2c1XvHx/view?usp=sharing
> 
> Python3 pickle file. Unigram mined based on the openwebtextcorpus linked above.

Hi @jhlau, it seems the link here has expired. It will be quite helpful if you can provide it again. Many thanks!<|||||>@GeassTaiga I no longer have it anymore unfortunately =/