readme: ---
  language: "ru"
  tags:
  - paraphrasing
  - russian
  license: mit
  ---
  
  This is a small Russian paraphraser based on the [google/mt5-small](https://huggingface.co/google/mt5-small) model. 
  It has rather poor paraphrasing performance, but can be fine tuned for this or other tasks.
  
  This model was created by taking the [alenusch/mt5small-ruparaphraser](https://huggingface.co/alenusch/mt5small-ruparaphraser) model and stripping 96% of its vocabulary which is unrelated to the Russian language or infrequent.
  
  * The original model has 300M parameters, with 256M of them being input and output embeddings. 
  * After shrinking the `sentencepiece` vocabulary from 250K to 20K the number of model parameters reduced to 65M parameters, and model size reduced from 1.1GB to 246MB.
     * The first 5K tokens in the new vocabulary are taken from the original `mt5-small`.
     * The next 15K tokens are the most frequent tokens obtained by tokenizing a Russian web corpus from the [Leipzig corpora collection](https://wortschatz.uni-leipzig.de/en/download/Russian).
  
  The model can be used as follows:
  ```
  # !pip install transformers sentencepiece
  import torch
  from transformers import T5ForConditionalGeneration, T5Tokenizer
  
  tokenizer = T5Tokenizer.from_pretrained("cointegrated/rut5-small")
  model = T5ForConditionalGeneration.from_pretrained("cointegrated/rut5-small")
  
  text = 'Ехал Грека через реку, видит Грека в реке рак. '
  inputs = tokenizer(text, return_tensors='pt')
  with torch.no_grad():
      hypotheses = model.generate(
          **inputs, 
          do_sample=True, top_p=0.95, num_return_sequences=10, 
          repetition_penalty=2.5,
          max_length=32,
      )
  for h in hypotheses:
      print(tokenizer.decode(h, skip_special_tokens=True))
  ```
  