title: Low-Rank Adapters Meet Neural Architecture Search for LLM Compression
summary: The rapid expansion of Large Language Models (LLMs) has posed significant
  challenges regarding the computational resources required for fine-tuning and
  deployment. Recent advancements in low-rank adapters have demonstrated their
  efficacy in parameter-efficient fine-tuning (PEFT) of these models. This
  retrospective paper comprehensively discusses innovative approaches that
  synergize low-rank representations with Neural Architecture Search (NAS)
  techniques, particularly weight-sharing super-networks. Robust solutions for
  compressing and fine-tuning large pre-trained models are developed by
  integrating these methodologies. Our analysis highlights the potential of these
  combined strategies to democratize the use of LLMs, making them more accessible
  for deployment in resource-constrained environments. The resulting models
  exhibit reduced memory footprints and faster inference times, paving the way
  for more practical and scalable applications of LLMs. Models and code are
  available at
  https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning.
paper_authors._id: 67999c3dc1e34886f90320ee
paper_authors.name: J. Pablo Muñoz
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-29T17:27:08.761Z
paper_authors._id: 63dece97e742e86dc92169b9
paper_authors.avatarUrl: /avatars/b408736f5089200ffd2898cd00132f0a.svg
paper_authors.fullname: J. Pablo Munoz
paper_authors.type: user
paper_authors.user: jpablomch
paper_authors._id: 67999c3dc1e34886f90320ef
paper_authors.name: Jinjie Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-29T17:27:14.404Z
paper_authors._id: 6541d41ae2e170b6a8de2f78
paper_authors.avatarUrl: /avatars/f7b524d17910b0e93548d08089d24f60.svg
paper_authors.fullname: Jinjie Yuan
paper_authors.type: user
paper_authors.user: jinjieyuan
paper_authors._id: 67999c3dc1e34886f90320f0
paper_authors.name: Nilesh Jain
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-29T17:27:20.569Z
paper_authors._id: 62d28f0c8d8206fd4d84ceca
paper_authors.avatarUrl: /avatars/bfcaa0ba84e80710964f4161e0aa56b7.svg
paper_authors.fullname: Nilesh Jain
paper_authors.type: user
paper_authors.user: dalle2
paper_title: Low-Rank Adapters Meet Neural Architecture Search for LLM Compression
paper_summary: The rapid expansion of Large Language Models (LLMs) has posed significant
  challenges regarding the computational resources required for fine-tuning and
  deployment. Recent advancements in low-rank adapters have demonstrated their
  efficacy in parameter-efficient fine-tuning (PEFT) of these models. This
  retrospective paper comprehensively discusses innovative approaches that
  synergize low-rank representations with Neural Architecture Search (NAS)
  techniques, particularly weight-sharing super-networks. Robust solutions for
  compressing and fine-tuning large pre-trained models are developed by
  integrating these methodologies. Our analysis highlights the potential of these
  combined strategies to democratize the use of LLMs, making them more accessible
  for deployment in resource-constrained environments. The resulting models
  exhibit reduced memory footprints and faster inference times, paving the way
  for more practical and scalable applications of LLMs. Models and code are
  available at
  https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning.
paper_ai_keywords: Large Language Models (LLMs) parameter-efficient fine-tuning (PEFT) low-rank adapters Neural Architecture Search (NAS) weight-sharing super-networks memory footprints inference times Hardware-Aware-Automated-Machine-Learning