title: LLM Augmented LLMs: Expanding Capabilities through Composition
summary: Foundational models with billions of parameters which have been trained on
  large corpora of data have demonstrated non-trivial skills in a variety of
  domains. However, due to their monolithic structure, it is challenging and
  expensive to augment them or impart new skills. On the other hand, due to their
  adaptation abilities, several new instances of these models are being trained
  towards new domains and tasks. In this work, we study the problem of efficient
  and practical composition of existing foundation models with more specific
  models to enable newer capabilities. To this end, we propose CALM --
  Composition to Augment Language Models -- which introduces cross-attention
  between models to compose their representations and enable new capabilities.
  Salient features of CALM are: (i) Scales up LLMs on new tasks by 're-using'
  existing LLMs along with a few additional parameters and data, (ii) Existing
  model weights are kept intact, and hence preserves existing capabilities, and
  (iii) Applies to diverse domains and settings. We illustrate that augmenting
  PaLM2-S with a smaller model trained on low-resource languages results in an
  absolute improvement of up to 13\% on tasks like translation into English and
  arithmetic reasoning for low-resource languages. Similarly, when PaLM2-S is
  augmented with a code-specific model, we see a relative improvement of 40\%
  over the base model for code generation and explanation tasks -- on-par with
  fully fine-tuned counterparts.
paper_authors._id: 659775ac40cffd3b17ad97e7
paper_authors.name: Rachit Bansal
paper_authors._id: 659775ac40cffd3b17ad97e8
paper_authors.name: Bidisha Samanta
paper_authors._id: 659775ac40cffd3b17ad97e9
paper_authors.name: Siddharth Dalmia
paper_authors._id: 659775ac40cffd3b17ad97ea
paper_authors.name: Nitish Gupta
paper_authors._id: 659775ac40cffd3b17ad97eb
paper_authors.name: Shikhar Vashishth
paper_authors._id: 659775ac40cffd3b17ad97ec
paper_authors.name: Sriram Ganapathy
paper_authors._id: 659775ac40cffd3b17ad97ed
paper_authors.name: Abhishek Bapna
paper_authors._id: 659775ac40cffd3b17ad97ee
paper_authors.name: Prateek Jain
paper_authors._id: 659775ac40cffd3b17ad97ef
paper_authors.name: Partha Talukdar
paper_title: LLM Augmented LLMs: Expanding Capabilities through Composition
paper_summary: Foundational models with billions of parameters which have been trained on
  large corpora of data have demonstrated non-trivial skills in a variety of
  domains. However, due to their monolithic structure, it is challenging and
  expensive to augment them or impart new skills. On the other hand, due to their
  adaptation abilities, several new instances of these models are being trained
  towards new domains and tasks. In this work, we study the problem of efficient
  and practical composition of existing foundation models with more specific
  models to enable newer capabilities. To this end, we propose CALM --
  Composition to Augment Language Models -- which introduces cross-attention
  between models to compose their representations and enable new capabilities.
  Salient features of CALM are: (i) Scales up LLMs on new tasks by 're-using'
  existing LLMs along with a few additional parameters and data, (ii) Existing
  model weights are kept intact, and hence preserves existing capabilities, and
  (iii) Applies to diverse domains and settings. We illustrate that augmenting
  PaLM2-S with a smaller model trained on low-resource languages results in an
  absolute improvement of up to 13\% on tasks like translation into English and
  arithmetic reasoning for low-resource languages. Similarly, when PaLM2-S is
  augmented with a code-specific model, we see a relative improvement of 40\%
  over the base model for code generation and explanation tasks -- on-par with
  fully fine-tuned counterparts.
paper_ai_summary: CALM, a composition method for large language models, enhances capabilities by integrating cross-attention with specialized models, demonstrating improvements on translation and coding tasks.
paper_ai_keywords: foundation models parameters large corpora adaptation abilities CALM Composition to Augment Language Models cross-attention LLMs PaLM2-S low-resource languages translation arithmetic reasoning code generation explanation tasks