5

Model, Hyperparameters, and Training Details We freeze the FLAN-T5-LARGE text encoder
in TANGO and only train the parameters of the latent diffusion model. The diffusion model is based
on the Stable Diffusion U-Net architecture [30, 31] and has a total of 866M parameters. We use 8
channels and a cross-attention dimension of 1024 in the U-Net model.
We use the AdamW optimizer [19] with a learning rate of 3e-5 and a linear learning rate scheduler
for training. We train the model for 40 epochs on the AudioCaps dataset and report results for the
checkpoint with the best validation loss, which we obtained at epoch 39. We use four A6000 GPUs
for training TANGO, where it takes a total of 52 hours to train 40 epochs, with validation at the end of
every epoch. We use a per GPU batch size of 3 (2 original + 1 augmented instance) with 4 gradient
accumulation steps. The effective batch size for training is 3 (instance) ∗4 (accumulation) ∗4 (GPU)
= 48.

3.2 Baseline Models
Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model