readme: ---
  language:
  - en
  tags:
  - audio
  - automatic-speech-recognition
  - hf-asr-leaderboard
  widget:
  - example_title: Librispeech sample 1
    src: https://cdn-media.huggingface.co/speech_samples/sample1.flac
  - example_title: Librispeech sample 2
    src: https://cdn-media.huggingface.co/speech_samples/sample2.flac
  model-index:
  - name: whisper-small.en
    results:
    - task:
        name: Automatic Speech Recognition
        type: automatic-speech-recognition
      dataset:
        name: LibriSpeech (clean)
        type: librispeech_asr
        config: clean
        split: test
        args:
          language: en
      metrics:
      - name: Test WER
        type: wer
        value: null
    - task:
        name: Automatic Speech Recognition
        type: automatic-speech-recognition
      dataset:
        name: LibriSpeech (other)
        type: librispeech_asr
        config: other
        split: test
        args:
          language: en
      metrics:
      - name: Test WER
        type: wer
        value: null
  pipeline_tag: automatic-speech-recognition
  license: apache-2.0
  duplicated_from: openai/whisper-small.en
  ---
  
  # Whisper 
  
  [OpenAI's Whisper](https://openai.com/blog/whisper/)
  
  The Whisper model was proposed in [Robust Speech Recognition via Large-Scale Weak Supervision](https://cdn.openai.com/papers/whisper.pdf) by Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, Ilya Sutskever.
  
  **Disclaimer**: Content from **this** model card has been written by the Hugging Face team, and parts of it were copy pasted from the original model card.
  
  
  ## Intro
  
  The first paragraphs of the abstract read as follows : 
  
  > We study the capabilities of speech processing systems trained simply to predict large amounts of transcripts of audio on the internet. When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zeroshot transfer setting without the need for any finetuning. 
  > When compared to humans, the models approach their accuracy and robustness. We are releasing models and inference code to serve as a foundation for further work on robust speech processing.
  
  The original code repository can be found [here](https://github.com/openai/whisper).
  
  ## Model details 
  
  The Whisper models are trained for speech recognition and translation tasks, capable of transcribing speech audio into the text in the language it is spoken (ASR) as well as translated into English (speech translation). Researchers at OpenAI developed the models to study the robustness of speech processing systems trained under large-scale weak supervision. There are 9 models of different sizes and capabilities, summarised in the following table.
  
  |  Size  | Parameters | English-only model | Multilingual model |  
  |:------:|:----------:|:------------------:|:------------------:|
  |  tiny  |    39 M    |         ✓          |         ✓          |
  |  base  |    74 M    |         ✓          |         ✓          |
  | small  |   244 M    |         ✓          |         ✓          |
  | medium |   769 M    |         ✓          |         ✓          |
  | large  |   1550 M   |                    |         ✓          |
  
  
  
  ## Model description 
  
  Whisper is an auto-regressive automatic speech recognition encoder-decoder model that was trained on 680 000 hours of 16kHz sampled multilingual audio. It was fully trained in a supervised manner, with multiple tasks : 
  
  - English transcription 
  - Any-to-English speech translation
  - Non-English transcription
  - No speech prediction 
  
  To each task corresponds a sequence of tokens that are given to the decoder as *context tokens*. The beginning of a transcription always starts with `<|startoftranscript|>` which is why the `decoder_start_token` is always set to `tokenizer.encode("<|startoftranscript|>")`. The following token should be the language token, which is automatically detected in the original code. Finally, the task is define using either `<|transcribe|>` or `<|translate|>`. In addition, a `<|notimestamps|>` token is added if the task does not include timestamp prediction.
  
  
  # Usage
  
  To transcribe or translate audio files, the model has to be used along a `WhisperProcessor`. The `WhisperProcessor.get_decoder_prompt_ids` function is used to get a list of `( idx, token )` tuples, which can either be set in the config, or directly passed to the generate function, as `forced_decoder_ids`. 
  
  
  ## Transcription 
  In the following example, the english only model is used. We set the `decoder_input_ids` accordingly.
  
  
  ### English to english 
  The "<|en|>" token is used to specify that the speech is in english and should be transcribed to english 
  
  ```python
  >>> from transformers import WhisperProcessor, WhisperForConditionalGeneration
  >>> from datasets import load_dataset
  >>> import torch
  
  >>> # load model and processor
  >>> processor = WhisperProcessor.from_pretrained("openai/whisper-small.en")
  >>> model = WhisperForConditionalGeneration.fro