pytorch fastspeech2 en dataset:LJSpeech dataset:LibriTTS arxiv:2006.04558 transformers audio TTS license:apache-2.0
.gitattributes README.md config.json pytorch_model.bin special_tokens_map.json tokenizer_config.json vocab.json
---
language: en
datasets:
- LJSpeech
- LibriTTS
tags:
- audio
- TTS
license: apache-2.0
---
# ontocord/fastspeech2-en
Modified version of the text-to-speech system [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech] (https://arxiv.org/abs/2006.04558v1).
## Installation
```
git clone https://github.com/ontocord/fastspeech2_hf
pip install transformers torchaudio

```
## Usage
The model can be used directly as follows:

```
# load the model and tokenizer
from fastspeech2_hf.modeling_fastspeech2 import FastSpeech2ForPretraining, FastSpeech2Tokenizer
model = FastSpeech2ForPretraining.from_pretrained("ontocord/fastspeech2-en")
tokenizer = FastSpeech2Tokenizer.from_pretrained("ontocord/fastspeech2-en")

# some helper routines
from IPython.display import Audio as IPAudio, display as IPdisplay
import torch
import torchaudio

def play_audio(waveform, sample_rate):
  waveform = waveform.numpy()
  if len(waveform.shape)==1:
    IPdisplay(IPAudio(waveform, rate=sample_rate))
    return 
  num_channels, num_frames = waveform.shape
  if num_channels <= 1:
    IPdisplay(IPAudio(waveform[0], rate=sample_rate))
  elif num_channels == 2:
    IPdisplay(IPAudio((waveform[0], waveform[1]), rate=sample_rate))
  else:
    raise ValueError("Waveform with more than 2 channels are not supported.")

# set the g2p module for the tokenizer
tokenizer.set_g2p(model.fastspeech2.g2p)
# you can run in half mode on gpu.
model = model.cuda().half()
sentences = [
        "Advanced text to speech models such as Fast Speech can synthesize speech significantly faster than previous auto regressive models with comparable quality. The training of Fast Speech model relies on an auto regressive teacher model for duration prediction and knowledge distillation, which can ease the one to many mapping problem in T T S. However, Fast Speech has several disadvantages, 1, the teacher student distillation pipeline is complicated, 2, the duration extracted from the teacher model is not accurate enough, and the target mel spectrograms distilled from teacher model suffer from information loss due to data simplification, both of which limit the voice quality. ",
        "Printing, in the only sense with which we are at present concerned, differs from most if not from all the arts and crafts represented in the Exhibition "
        "in being comparatively modern. ",
        "For although the Chinese took impressions from wood blocks engraved in relief for centuries before the woodcutters of the Netherlands, by a similar process "
        "produced the block books, which were the immediate predecessors of the true printed book, "
        "the invention of movable metal letters in the middle of the fifteenth century may justly be considered as the invention of the art of printing. ",
        "And it is worth mention in passing that, as an example of fine typography, "
        "the earliest book printed with movable types, the Gutenberg, or \"forty-two line Bible\" of about 1455, "
        "has never been surpassed. ",
        "Printing, then, for our purpose, may be considered as the art of making books by means of movable types. "
        "Now, as all books not primarily intended as picture-books consist principally of types composed to form letterpress,",
       ]
batch = tokenizer(sentences, return_tensors="pt", padding=True)

model.eval()
with torch.no_grad():
  out = model(use_postnet=False, **batch)
wav =out[-2]
for line, phone, w in zip(sentences, tokenizer.batch_decode(batch['input_ids']), wav):
  print ("txt:", line)
  print ("phoneme:", phone)
  play_audio(w.type(torch.FloatTensor), model.config.sampling_rate)


```

##Github Code Repo 
Current code for this model can be found [here](https://github.com/ontocord/fastspeech2_hf)

This is a work in progress (WIP) port of the model and code from  
[this repo] (https://github.com/ming024/FastSpeech2).

The datasets on which this model was trained:
- LJSpeech: a single-speaker English dataset consists of 13100 short audio clips of a female speaker reading passages from 7 non-fiction books, approximately 24 hours in total.
- LibriTTS: a multi-speaker English dataset containing 585 hours of speech by 2456 speakers.

