Audio VAE and Vocoder. We use the audio VAE model from Liu et al. [17]. This VAE net-
work was trained on the AudioSet, AudioCaps, Freesound2, and BBC Sound Effect Library3 (SFX)
datasets. Longer audio clips in Freesound and BBC SFX were truncated to the ﬁrst thirty seconds
and then segmented into three parts of ten seconds each. All audio clips were resampled in 16KHz
frequency for training the VAE network. We used a compression level of 4 with 8 latent channels
for the VAE network.
We also use the vocoder from Liu et al. [17] for audio waveform generation from the mel spec-
trogram generated by the VAE decoder. The vocoder is a HiFi-GAN [13] network trained on the
AudioSet dataset. All audio clips were resampled at 16KHz for training the vocoder network.

2https://freesound.org/
3https://sound-effects.bbcrewind.co.uk

5
Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model