readme: ---
  language: ru
  datasets:
  - bond005/sberdevices_golos_10h_crowd
  - bond005/sberdevices_golos_100h_farfield
  - common_voice
  - bond005/sova_rudevices
  metrics:
  - wer
  - cer
  tags:
  - audio
  - automatic-speech-recognition
  - speech
  - common_voice
  - SberDevices/Golos
  - sova_rudevices
  license: apache-2.0
  widget:
  - example_title: test sound with Russian speech
    src: https://huggingface.co/bond005/wav2vec2-mbart50-ru/resolve/main/test_sound.wav
  model-index:
  - name: Wav2Vec2-mBART-50 for speech-to-text in Russian by Ivan Bondarenko
    results:
    - task:
        name: Speech Recognition
        type: automatic-speech-recognition
      dataset:
        name: Sberdevices Golos (crowd)
        type: SberDevices/Golos
        args: ru
      metrics:
         - name: Test WER
           type: wer
           value: 14.553
         - name: Test CER
           type: cer
           value: 5.327
    - task:
        name: Speech Recognition
        type: automatic-speech-recognition
      dataset:
        name: Sberdevices Golos (farfield)
        type: SberDevices/Golos
        args: ru
      metrics:
         - name: Test WER
           type: wer
           value: 21.403
         - name: Test CER
           type: cer
           value: 9.756
    - task:
        name: Automatic Speech Recognition
        type: automatic-speech-recognition
      dataset:
        name: Common Voice ru
        type: common_voice
        args: ru
      metrics:
      - name: Test WER
        type: wer
        value: 15.891
      - name: Test CER
        type: cer
        value: 6.136
    - task:
        name: Speech Recognition
        type: automatic-speech-recognition
      dataset:
        name: Sova RuDevices
        type: sova_rudevices
        args: ru
      metrics:
      - name: Test WER
        type: wer
        value: 25.771
      - name: Test CER
        type: cer
        value: 12.037
  ---
  # Wav2Vec2-mBART-50-Ru
  
  Wav2Vec2-mBART-50-Ru is a speech-sequence-to-text-sequence model, which can convert an input audio with Russian speech into a text with punctuation, capitalization and so on.
  
  Wav2Vec2-mBART-50-Ru is the [SpeechEncoderDecoderModel](https://huggingface.co/docs/transformers/model_doc/speech-encoder-decoder), which was initialized with [Wav2Vec2-Large-Ru-Golos](https://huggingface.co/bond005/wav2vec2-large-ru-golos) as the encoder and [mBART-large-50](https://huggingface.co/facebook/mbart-large-50) as the decoder. After its initialization the model was fine-tuned using the training parts of several annotated speech corpora: 
  
  - [the 10 hours crowd subset of SberDevices Golos](https://huggingface.co/datasets/bond005/sberdevices_golos_10h_crowd)
  - [the 100 hours farfield subset of SberDevices Golos](https://huggingface.co/datasets/bond005/sberdevices_golos_100h_farfield)
  - [the Russian subset of Common Voice 6.0](https://huggingface.co/datasets/common_voice)
  - [Sova RuDevices](https://huggingface.co/datasets/bond005/sova_rudevices)
  
  CommonVoice 6.0 contains "rich" text annotations with punctuation and capitalization, but other speech corpora includes plain texts only. Therefore, text annotations of these corpora were riched automatically using the [Silero text enhancement model](https://github.com/snakers4/silero-models#text-enhancement).
  
  ## Usage
  
  When using this model, make sure that your speech input is sampled at 16kHz.
  
  You can use this model by writing your own inference script:
  
  ```python
  import os
  import warnings
  
  import torch
  from datasets import load_dataset
  from datasets.features import Audio
  from transformers import SpeechEncoderDecoderModel, Wav2Vec2Processor
  
  LANG_ID = "ru"
  MODEL_ID = "bond005/wav2vec2-mbart50-ru"
  SAMPLES = 32
  
  num_processes = max(1, os.cpu_count())
  
  processor = Wav2Vec2Processor.from_pretrained(MODEL_ID)
  model = SpeechEncoderDecoderModel.from_pretrained(MODEL_ID)
  
  test_dataset = load_dataset("common_voice", LANG_ID, split=f"test[:{SAMPLES}]")
  
  if test_dataset.features['audio'].sampling_rate != 16_000:
      test_dataset = test_dataset.cast_column(
          'audio',
          Audio(sampling_rate=16_000)
      )
  
  audio_data = [test_dataset[i]['audio']['array'] for i in range(SAMPLES)]
  
  processed = processor(audio_data, sampling_rate=16_000,
                        return_tensors="pt", padding='longest')
  
  with torch.no_grad():
      predicted_ids = model.generate(**processed)
  
  predicted_sentences = processor.batch_decode(
      predicted_ids,
      num_processes=num_processes,
      skip_special_tokens=True
  )
  
  with warnings.catch_warnings():
      warnings.simplefilter("ignore")
      for i, predicted_sentence in enumerate(predicted_sentences):
          print("-" * 100)
          print("Reference: ", test_dataset[i]["sentence"])
          print("Prediction:", predicted_sentence)
  ```
  
  ```text
  ----------------------------------------------------------------------------------------------------
  Reference:  Я беру маленький кусочек бумажки.
  Prediction: Я беру маленькие кусочки бумажки.
  ----------------------------------------------------------------------------------------------------
  Reference:  О потерях пока не сообщается.
  Prediction: А потеря и