readme: ---
  license: apache-2.0
  language:
  - en
  pipeline_tag: automatic-speech-recognition
  datasets:
  - LRS3
  
  tags:
  - Audio Visual to Text
  - Automatic Speech Recognition
  ---
  
  ## Model Description
  
  These are model weights originally provided by the authors of the paper [Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction](https://arxiv.org/pdf/2201.02184.pdf).
  
  <figure>
    <img src="https://huggingface.co/vumichien/AV-HuBERT/resolve/main/HuBert.png" alt="Audio-visual HuBERT">
    <figcaption>Audio-visual HuBERT
    </figcaption>
  </figure>
  
  Video recordings of speech contain correlated audio and visual information, providing a strong signal for speech representation learning from the speaker’s lip
  movements and the produced sound. 
  
  Audio-Visual Hidden Unit BERT (AV-HuBERT), a self-supervised representation learning framework for audio-visual speech, which masks multi-stream video input and predicts automatically discovered and iteratively refined multimodal hidden units. AV-HuBERT
  learns powerful audio-visual speech representation benefiting both lip-reading and automatic speech recognition.
  
  The official code of this paper in [here](https://github.com/facebookresearch/av_hubert)
  
  ## Example
  
  <figure>
    <img src="https://huggingface.co/vumichien/AV-HuBERT/resolve/main/lipreading.gif" alt="Audio-Visual Speech Recognition">
    <figcaption> Speech Recognition from visual lip movement
    </figcaption>
  </figure>
  
  ## Datasets
  The authors trained the model on LRS3 with 433 hours of transcribed English videos and English portion of VoxCeleb2, which amounts to 1,326 hours