While our proposal is agnostic to training objective, we focus on language modeling as our motivat-
ing use case. Below is a brief description of the language modeling problem and, in particular, the
maximization of conditional probabilities given a task-speciﬁc prompt.
Suppose we are given a pre-trained autoregressive language model PΦ(y|x) parametrized by Φ.
For instance, PΦ(y|x) can be a generic multi-task learner such as GPT (Radford et al., b; Brown
et al., 2020) based on the Transformer architecture (Vaswani et al., 2017). Consider adapting this
pre-trained model to downstream conditional text generation tasks, such as summarization, machine
reading comprehension (MRC), and natural language to SQL (NL2SQL). Each downstream task is
represented by a training dataset of context-target pairs: Z = {(xi, yi)}i=1,..,N , where both xi and
yi are sequences of tokens. For example, in NL2SQL, xi is a natural language query and yi its
LORA