3 Experiments

3.1 Datasets and Training

Text-to-Audio Generation. We perform our main text-to-audio generation experiments on the
AudioCaps dataset [11]. The dataset contains 45,438 audio clips paired with human-written captions
for training. The validation set contains 2,240 instances. The audio clips are ten seconds long
and were collected from YouTube videos. The clips were originally crowd-sourced as part of the
signiﬁcantly larger AudioSet dataset [4] for the audio classiﬁcation task.
We train our LDM using only the paired (text, audio) instances from the AudioCaps dataset. We
use the AudioCaps test set as our evaluation data. The test set contains ﬁve human-written captions
for each audio clip. We use one caption for each clip chosen at random following Liu et al. [17]
for consistent evaluation with their work. The randomly chosen caption is used as the text prompt,
using which we generate the audio signal from our model.
Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model