title: Towards Modular LLMs by Building and Reusing a Library of LoRAs
summary: The growing number of parameter-efficient adaptations of a base large
  language model (LLM) calls for studying whether we can reuse such trained
  adapters to improve performance for new tasks. We study how to best build a
  library of adapters given multi-task data and devise techniques for both
  zero-shot and supervised task generalization through routing in such library.
  We benchmark existing approaches to build this library and introduce
  model-based clustering, MBC, a method that groups tasks based on the similarity
  of their adapter parameters, indirectly optimizing for transfer across the
  multi-task dataset. To re-use the library, we present a novel zero-shot routing
  mechanism, Arrow, which enables dynamic selection of the most relevant adapters
  for new inputs without the need for retraining. We experiment with several
  LLMs, such as Phi-2 and Mistral, on a wide array of held-out tasks, verifying
  that MBC-based adapters and Arrow routing lead to superior generalization to
  new tasks. We make steps towards creating modular, adaptable LLMs that can
  match or outperform traditional joint training.
paper_authors._id: 664c0bf1cc9c64354ac9478e
paper_authors.name: Oleksiy Ostapenko
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:03:15.164Z
paper_authors._id: 63be75e724e75d1efdf6ae36
paper_authors.avatarUrl: /avatars/62d031d7fdb0d0747476d52c78dcdb18.svg
paper_authors.fullname: Oleksiy Ostapenko
paper_authors.type: user
paper_authors.user: ostapeno
paper_authors._id: 664c0bf1cc9c64354ac9478f
paper_authors.name: Zhan Su
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:03:36.301Z
paper_authors._id: 64dbf68ce1e953158d591adb
paper_authors.avatarUrl: /avatars/30f48e9baaf2fbc7a9b0412a99b50bf2.svg
paper_authors.fullname: Zhan Su
paper_authors.type: user
paper_authors.user: zhan1993
paper_authors._id: 664c0bf1cc9c64354ac94790
paper_authors.name: Edoardo Maria Ponti
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:03:42.851Z
paper_authors._id: 60809ad44ad99100d63ce36a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1619040921084-noauth.jpeg
paper_authors.fullname: Edoardo Maria Ponti
paper_authors.type: user
paper_authors.user: ducdauge
paper_authors._id: 664c0bf1cc9c64354ac94791
paper_authors.name: Laurent Charlin
paper_authors._id: 664c0bf1cc9c64354ac94792
paper_authors.name: Nicolas Le Roux
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:03:57.607Z
paper_authors._id: 6380eaa5b2552a70c4a19205
paper_authors.avatarUrl: /avatars/99f601bf922996a51fe9782cd20afd28.svg
paper_authors.fullname: Nicolas Le Roux
paper_authors.type: user
paper_authors.user: nicolouchka
paper_authors._id: 664c0bf1cc9c64354ac94793
paper_authors.name: Matheus Pereira
paper_authors._id: 664c0bf1cc9c64354ac94794
paper_authors.name: Lucas Caccia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:04:31.941Z
paper_authors._id: 63cde3b2ff7cd335f0ea4441
paper_authors.avatarUrl: /avatars/7a067499b7141efb4f2e7eb97d3807d5.svg
paper_authors.fullname: Lucas Caccia
paper_authors.type: user
paper_authors.user: pclucas14
paper_authors._id: 664c0bf1cc9c64354ac94795
paper_authors.name: Alessandro Sordoni
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-05-21T02:53:08.598Z
paper_authors._id: 637c27e5e74a3e06aacfd7b4
paper_authors.avatarUrl: /avatars/27dcca787b11d880c0ad851e83a84e87.svg
paper_authors.fullname: Alessandro Sordoni
paper_authors.type: user
paper_authors.user: sordonia
paper_title: Towards Modular LLMs by Building and Reusing a Library of LoRAs
paper_summary: The growing number of parameter-efficient adaptations of a base large
  language model (LLM) calls for studying whether we can reuse such trained
  adapters to improve performance for new tasks. We study how to best build a
  library of adapters given multi-task data and devise techniques for both
  zero-shot and supervised task generalization through routing in such library.
  We benchmark existing approaches to build this library and introduce
  model-based clustering, MBC, a method that groups tasks based on the similarity
  of their adapter parameters, indirectly optimizing for transfer across the
  multi-task dataset. To re-use the library, we present a novel zero-shot routing
  mechanism, Arrow, which enables dynamic selection of the most relevant adapters
  for new inputs without the need for retraining. We experiment with several
  LLMs, such as Phi-2 and Mistral, on a wide array of held-out tasks, verifying
  that MBC-based adapters and Arrow routing lead to superior generalization to
  new tasks. We make steps towards creating modular, adaptable LLMs that can
  match or outperform traditional joint training.
paper_ai_summary: A novel approach using model-based clustering and zero-shot routing improves the generalization of adapters in large language models for new tasks.
paper_ai_keywords: parameter-efficient adaptations large language model (LLM) adapters multi-task data zero-shot task generalization supervised task generalization model-based clustering MBC task similarity transfer learning zero-shot routing Arrow dynamic selection joint training