readme: ---
  language:
  - ru
  - myv
  tags:
  - erzya
  - mordovian
  - fill-mask
  - pretraining
  - embeddings
  - masked-lm
  - feature-extraction
  - sentence-similarity
  license: cc-by-sa-4.0
  datasets:
    - slone/myv_ru_2022
  ---
  
  This is an Erzya (`myv`, cyrillic script) sentence encoder from the paper [The first neural machine translation system for the Erzya language](https://arxiv.org/abs/2209.09368).
  
  It is based on [sentence-transformers/LaBSE](https://huggingface.co/sentence-transformers/LaBSE) ([license here](https://tfhub.dev/google/LaBSE/2)), but with updated vocabulary and checkpoint:
  - Removed all tokens except the most popular ones for English or Russian;
  - Added extra tokens for Erzya language;
  - Fine-tuned on the [slone/myv_ru_2022](https://huggingface.co/slone/myv_ru_2022) corpus using a mixture of tasks:
    - Cross-lingual distillation of sentence embeddings from the original LaBSE model, using the parallel `ru-myv` corpus;
    - Masked language modelling on `myv` monolingual data;
    - Sentence pair classification to distinguish correct `ru-myv` translations from random pairs.
    
   The model can be used as a sentence encoder or a masked language modelling predictor for Erzya, or fine-tuned for any downstream NLU dask.
   
   Sentence embeddings can be produced with the code below:
   ```python
  import torch
  from transformers import AutoTokenizer, AutoModel
  tokenizer = AutoTokenizer.from_pretrained("slone/LaBSE-en-ru-myv-v1")
  model = AutoModel.from_pretrained("slone/LaBSE-en-ru-myv-v1")
  sentences = ["Hello World", "Привет Мир", "Шумбратадо Мастор"]
  encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
  with torch.no_grad():
      model_output = model(**encoded_input)
  embeddings = model_output.pooler_output
  embeddings = torch.nn.functional.normalize(embeddings)
  print(embeddings.shape)  # torch.Size([3, 768])
  ```
  
  
  