readme: ---
  language: ["ru"]
  tags:
  - russian
  license: mit
  ---
  
  This is the [rut5-base](https://huggingface.co/cointegrated/rut5-base) model, with the decoder fine-tuned to recover (approximately) Russian sentences from their [LaBSE](https://huggingface.co/sentence-transformers/LaBSE) embeddings. Details are [here](https://habr.com/ru/post/677618/) (in Russian).
  
  It can be used, for example, for:
  - Paraphrasing Russian sentences;
  - Translating from the 109 LaBSE languages to Russian;
  - Summarizing a collection of sentences with a single sentence;
  - Interpolating between sentences;
  - Few-shot text style transfer (including cross-lingual).
  
  Example code:
  ```python
  import torch
  from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, AutoModel
  from transformers.modeling_outputs import BaseModelOutput
  
  enc_tokenizer = AutoTokenizer.from_pretrained('cointegrated/LaBSE-en-ru')
  encoder = AutoModel.from_pretrained('cointegrated/LaBSE-en-ru')
  
  dec_tokenizer = AutoTokenizer.from_pretrained('cointegrated/rut5-base-labse-decoder')
  decoder = AutoModelForSeq2SeqLM.from_pretrained('cointegrated/rut5-base-labse-decoder')
  
  def encode(texts):
      encoded_input = enc_tokenizer(texts, padding=True, truncation=True, max_length=512, return_tensors='pt')
      with torch.no_grad():
          model_output = encoder(**encoded_input.to(encoder.device))
          embeddings = model_output.pooler_output
          embeddings = torch.nn.functional.normalize(embeddings)
      return embeddings
   
  # encode some texts into vectors
  embeddings = encode([
      "4 декабря 2000 года",
      "Давно такого не читала, очень хорошо пишешь!",
      "Я тогда не понимала, что происходит, не понимаю и сейчас.",
      "London is the capital of Great Britain.",
  ])
  print(embeddings.shape)
  # torch.Size([4, 768])
  
  # now try to recover the texts from the vectors
  out = decoder.generate(
      encoder_outputs=BaseModelOutput(last_hidden_state=embeddings.unsqueeze(1)), 
      max_length=256, 
      repetition_penalty=3.0,
  )
  for tokens in out:
      print(dec_tokenizer.decode(tokens, skip_special_tokens=True))
  # После 4 декабря 2000 года
  # Не так давно, это многое читала!
  # Я не понимала того, что происходит сейчас тогда, дальше.
  # Британская столица Англии.
  ```