readme: ---
  language: en
  license: apache-2.0
  tags:
    - phoneme-recognition
    - generated_from_trainer
  datasets:
    - w11wo/ljspeech_phonemes
  model-index:
    - name: Wav2Vec2 LJSpeech Gruut
      results:
        - task:
            name: Automatic Speech Recognition
            type: automatic-speech-recognition
          dataset:
            name: LJSpeech
            type: ljspeech_phonemes
          metrics:
            - name: Test PER (w/o stress)
              type: per
              value: 0.0099
            - name: Test CER (w/o stress)
              type: cer
              value: 0.0058
  ---
  
  # Wav2Vec2 LJSpeech Gruut
  
  Wav2Vec2 LJSpeech Gruut is an automatic speech recognition model based on the [wav2vec 2.0](https://arxiv.org/abs/2006.11477) architecture. This model is a fine-tuned version of [Wav2Vec2-Base](https://huggingface.co/facebook/wav2vec2-base) on the [LJSpech Phonemes](https://huggingface.co/datasets/w11wo/ljspeech_phonemes) dataset.
  
  Instead of being trained to predict sequences of words, this model was trained to predict sequence of phonemes, e.g. `["h", "ɛ", "l", "ˈoʊ", "w", "ˈɚ", "l", "d"]`. Therefore, the model's [vocabulary](https://huggingface.co/bookbot/wav2vec2-ljspeech-gruut/blob/main/vocab.json) contains the different IPA phonemes found in [gruut](https://github.com/rhasspy/gruut).
  
  This model was trained using HuggingFace's PyTorch framework. All training was done on a Google Cloud Engine VM with a Tesla A100 GPU. All necessary scripts used for training could be found in the [Files and versions](https://huggingface.co/bookbot/wav2vec2-ljspeech-gruut/tree/main) tab, as well as the [Training metrics](https://huggingface.co/bookbot/wav2vec2-ljspeech-gruut/tensorboard) logged via Tensorboard.
  
  ## Model
  
  | Model                     | #params | Arch.       | Training/Validation data (text) |
  | ------------------------- | ------- | ----------- | ------------------------------- |
  | `wav2vec2-ljspeech-gruut` | 94M     | wav2vec 2.0 | `LJSpech Phonemes` Dataset      |
  
  ## Evaluation Results
  
  The model achieves the following results on evaluation:
  
  | Dataset                      | PER (w/o stress) | CER (w/o stress) |
  | ---------------------------- | :--------------: | :--------------: |
  | `LJSpech Phonemes` Test Data |      0.99%       |      0.58%       |
  
  ## Usage
  
  ```py
  from transformers import AutoProcessor, AutoModelForCTC, Wav2Vec2Processor
  import librosa
  import torch
  from itertools import groupby
  from datasets import load_dataset
  
  def decode_phonemes(
      ids: torch.Tensor, processor: Wav2Vec2Processor, ignore_stress: bool = False
  ) -> str:
      """CTC-like decoding. First removes consecutive duplicates, then removes special tokens."""
      # removes consecutive duplicates
      ids = [id_ for id_, _ in groupby(ids)]
  
      special_token_ids = processor.tokenizer.all_special_ids + [
          processor.tokenizer.word_delimiter_token_id
      ]
      # converts id to token, skipping special tokens
      phonemes = [processor.decode(id_) for id_ in ids if id_ not in special_token_ids]
  
      # joins phonemes
      prediction = " ".join(phonemes)
  
      # whether to ignore IPA stress marks
      if ignore_stress == True:
          prediction = prediction.replace("ˈ", "").replace("ˌ", "")
  
      return prediction
  
  checkpoint = "bookbot/wav2vec2-ljspeech-gruut"
  
  model = AutoModelForCTC.from_pretrained(checkpoint)
  processor = AutoProcessor.from_pretrained(checkpoint)
  sr = processor.feature_extractor.sampling_rate
  
  # load dummy dataset and read soundfiles
  ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean", split="validation")
  audio_array = ds[0]["audio"]["array"]
  
  # or, read a single audio file
  # audio_array, _ = librosa.load("myaudio.wav", sr=sr)
  
  inputs = processor(audio_array, return_tensors="pt", padding=True)
  
  with torch.no_grad():
      logits = model(inputs["input_values"]).logits
  
  predicted_ids = torch.argmax(logits, dim=-1)
  prediction = decode_phonemes(predicted_ids[0], processor, ignore_stress=True)
  # => should give 'b ɪ k ʌ z j u ɚ z s l i p ɪ ŋ ɪ n s t ɛ d ə v k ɔ ŋ k ɚ ɪ ŋ ð ə l ʌ v l i ɹ z p ɹ ɪ n s ə s h æ z b ɪ k ʌ m ə v f ɪ t ə l w ɪ θ n b oʊ p ɹ ə ʃ æ ɡ i s ɪ t s ð ɛ ɹ ə k u ɪ ŋ d ʌ v'
  ```
  
  ## Training procedure
  
  ### Training hyperparameters
  
  The following hyperparameters were used during training:
  
  - `learning_rate`: 0.0001
  - `train_batch_size`: 16
  - `eval_batch_size`: 8
  - `seed`: 42
  - `gradient_accumulation_steps`: 2
  - `total_train_batch_size`: 32
  - `optimizer`: Adam with `betas=(0.9,0.999)` and `epsilon=1e-08`
  - `lr_scheduler_type`: linear
  - `lr_scheduler_warmup_steps`: 1000
  - `num_epochs`: 30.0
  - `mixed_precision_training`: Native AMP
  
  ### Training results
  
  | Training Loss | Epoch | Step  | Validation Loss |  Wer   |  Cer   |
  | :-----------: | :---: | :---: | :-------------: | :----: | :----: |
  |    No log     |  1.0  |  348  |     2.2818      |  1.0   |  1.0   |
  |    2.6692     |  2.0  |  696  |     0.2045      | 0.0527 | 0.0299 |
  |    0.2225     |  3.0  | 1044  |     0.1162      | 0.0319 | 0.0189 |
  |    0.2225