We use the pre-trained LLM FLAN-T5-LARGE (780M) [2] as the text encoder (Etext) to obtain
text encoding τ ∈ RL×dtext, where L and dtext are the token count and token-embedding size,
respectively. Due to the pre-training of FLAN-T5 models on a large-scale chain-of-thought- (CoT)
and instruction-based dataset, Dai et al. [3] posit that they are able to learn a new task very well
from the in-context information by mimicking gradient descent through attention weights. This
property is missing in the older large models, such as RoBERTa [18] (used by Liu et al. [17]) and
T5 [27] (used by Kreuk et al. [16]). Considering each input sample a distinct task, it might be
reasonable to assume that the gradient-descent mimicking property could be pivotal in learning the
mapping between textual and acoustic concepts without ﬁne-tuning the text encoder. The richer pre-
training may also allow the encoder to better emphasize the key details with less noise and enriched
Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model