process, as in how much information is lost when encoding waveform into the features, and the
decoding process, as in how well the vocoder can translate the encoded information into waveform.
To motivate the choice of the acoustic feature and the vocoder, we compare four combinations:
the first one is Mel spectrogram + HiFi-GAN which is what this paper adopts. The second is Mel
spectrogram + Parallel WaveGAN [Yamamoto et al., 2020] that is used by A3T [Bai et al., 2022].
The third one is Encodec post-quantization dense feature + Encodec decoder, which is analogous to
VALL-E’s setup. The last one is also Encodec but with pre-quantization dense feature, which we
include to study how much information is lost during quantization.
We also note that Mel spectrogram features are 80 dimensional encoded at 100Hz, which is 8K
dimensions per second, while Encodec features are 128 dimensional encoded at 75Hz, which is 9.6K
dimensions per second, higher than the Mel spectrogram features.
Voicebox-Text-GuidedMultilingual UniversalSpeechGenerationatScale