readme: ---
  language: 
    - es
    - af
    - ar
    - arz
    - as
    - bn
    - fr
    - sw
    - eu
    - ca
    - zh
    - en
    - hi
    - ur
    - id
    - pt
    - vi
    - gu
    - kn
    - ml
    - mr
    - ta
    - te
    - yo
  tags:
  - kenlm
  - perplexity
  - n-gram
  - kneser-ney
  - bigscience
  license: "mit"
  datasets:
  - wikipedia
  - oscar
  ---
  
  # KenLM models
  This repo contains several KenLM models trained on different tokenized datasets and languages.  
  KenLM models are probabilistic n-gram languge models that models. One use case of these models consist on fast perplexity estimation for [filtering or sampling large datasets](https://huggingface.co/bertin-project/bertin-roberta-base-spanish). For example, one could use a KenLM model trained on French Wikipedia to run inference on a large dataset and filter out samples that are very unlike to appear on Wikipedia (high perplexity), or very simple non-informative sentences that could appear repeatedly (low perplexity).
  
  At the root of this repo you will find different directories named after the dataset models were trained on (e.g. `wikipedia`, `oscar`). Within each directory, you will find several models trained on different language subsets of the dataset (e.g. `en (English)`, `es (Spanish)`, `fr (French)`). For each language you will find three different files
  * `{language}.arpa.bin`: The trained KenLM model binary
  * `{language}.sp.model`: The trained SentencePiece model used for tokenization
  * `{language}.sp.vocab`: The vocabulary file for the SentencePiece model
  
  The models have been trained using some of the preprocessing steps from [cc_net](https://github.com/facebookresearch/cc_net), in particular replacing numbers with zeros and normalizing punctuation. So, it is important to keep the default values for the parameters: `lower_case`, `remove_accents`, `normalize_numbers` and `punctuation` when using the pre-trained models in order to replicate the same pre-processing steps at inference time.
  
  # Dependencies
  * KenLM: `pip install https://github.com/kpu/kenlm/archive/master.zip`
  * SentencePiece: `pip install sentencepiece`
  
  # Example:
  ```
  from model import KenlmModel
  
  
  # Load model trained on English wikipedia
  model = KenlmModel.from_pretrained("wikipedia", "en")
  
  # Get perplexity
  model.get_perplexity("I am very perplexed")
  # 341.3 (low perplexity, since sentence style is formal and with no grammar mistakes)
  
  model.get_perplexity("im hella trippin")
  # 46793.5 (high perplexity, since the sentence is colloquial and contains grammar mistakes)
  ```
  In the example above we see that, since Wikipedia is a collection of encyclopedic articles, a KenLM model trained on it will naturally give lower perplexity scores to sentences with formal language and no grammar mistakes than colloquial sentences with grammar mistakes.