readme: ---
  library_name: fairseq
  task: audio-to-audio
  tags:
  - fairseq
  - audio
  - audio-to-audio
  - speech-to-speech-translation
  
  datasets:
  - mtedx
  - covost2
  - europarl_st
  - voxpopuli
  widget:
  - example_title: Common Voice sample 1
    src: https://huggingface.co/facebook/xm_transformer_600m-es_en-multi_domain/resolve/main/common_voice_es_19966634.flac
  ---
  ## xm_transformer_s2ut_800m-es-en-st-asr-bt_h1_2022
  
  Speech-to-speech translation model from fairseq S2UT ([paper](https://arxiv.org/abs/2204.02967)/[code](https://github.com/facebookresearch/fairseq/blob/main/examples/speech_to_speech/docs/enhanced_direct_s2st_discrete_units.md)):
  - Spanish-English
  - Trained on mTEDx, CoVoST 2, Europarl-ST and VoxPopuli
  - Speech synthesis with [facebook/unit_hifigan_mhubert_vp_en_es_fr_it3_400k_layer11_km1000_lj_dur](https://huggingface.co/facebook/unit_hifigan_mhubert_vp_en_es_fr_it3_400k_layer11_km1000_lj_dur)
  
  ## Usage
  ```python
  import json
  import os
  from pathlib import Path
  
  import IPython.display as ipd
  from fairseq import hub_utils
  from fairseq.checkpoint_utils import load_model_ensemble_and_task_from_hf_hub
  from fairseq.models.speech_to_text.hub_interface import S2THubInterface
  from fairseq.models.text_to_speech import CodeHiFiGANVocoder
  from fairseq.models.text_to_speech.hub_interface import VocoderHubInterface
  
  from huggingface_hub import snapshot_download
  import torchaudio
  
  cache_dir = os.getenv("HUGGINGFACE_HUB_CACHE")
  
  models, cfg, task = load_model_ensemble_and_task_from_hf_hub(
      "facebook/xm_transformer_s2ut_800m-es-en-st-asr-bt_h1_2022",
      arg_overrides={"config_yaml": "config.yaml", "task": "speech_to_text"},
      cache_dir=cache_dir,
  )
  #model = models[0].cpu()
  #cfg["task"].cpu = True
  generator = task.build_generator([model], cfg)
  
  
  # requires 16000Hz mono channel audio
  audio, _ = torchaudio.load("/path/to/an/audio/file")
  
  sample = S2THubInterface.get_model_input(task, audio)
  unit = S2THubInterface.get_prediction(task, model, generator, sample)
  
  # speech synthesis           
  library_name = "fairseq"
  cache_dir = (
      cache_dir or (Path.home() / ".cache" / library_name).as_posix()
  )
  cache_dir = snapshot_download(
      f"facebook/unit_hifigan_mhubert_vp_en_es_fr_it3_400k_layer11_km1000_lj_dur", cache_dir=cache_dir, library_name=library_name
  )
  
  x = hub_utils.from_pretrained(
      cache_dir,
      "model.pt",
      ".",
      archive_map=CodeHiFiGANVocoder.hub_models(),
      config_yaml="config.json",
      fp16=False,
      is_vocoder=True,
  )
  
  with open(f"{x['args']['data']}/config.json") as f:
      vocoder_cfg = json.load(f)
  assert (
      len(x["args"]["model_path"]) == 1
  ), "Too many vocoder models in the input"
  
  vocoder = CodeHiFiGANVocoder(x["args"]["model_path"][0], vocoder_cfg)
  tts_model = VocoderHubInterface(vocoder_cfg, vocoder)
  
  tts_sample = tts_model.get_model_input(unit)
  wav, sr = tts_model.get_prediction(tts_sample)
  
  ipd.Audio(wav, rate=sr)
  ```
  
  ## Citation
  ```bibtex
  @misc{https://doi.org/10.48550/arxiv.2204.02967,
    doi = {10.48550/ARXIV.2204.02967},  
    url = {https://arxiv.org/abs/2204.02967},  
    author = {Popuri, Sravya and Chen, Peng-Jen and Wang, Changhan and Pino, Juan and Adi, Yossi and Gu, Jiatao and Hsu, Wei-Ning and Lee, Ann},  
    keywords = {Computation and Language (cs.CL), Sound (cs.SD), Audio and Speech Processing (eess.AS), FOS: Computer and information sciences, FOS: Computer and information sciences, FOS: Electrical engineering, electronic engineering, information engineering, FOS: Electrical engineering, electronic engineering, information engineering},  
    title = {Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation},  
    publisher = {arXiv},  
    year = {2022},  
    copyright = {arXiv.org perpetual, non-exclusive license}
  }
  ```