title: LLaMA-NAS: Efficient Neural Architecture Search for Large Language
    Models
summary: The abilities of modern large language models (LLMs) in solving natural
  language processing, complex reasoning, sentiment analysis and other tasks have
  been extraordinary which has prompted their extensive adoption. Unfortunately,
  these abilities come with very high memory and computational costs which
  precludes the use of LLMs on most hardware platforms. To mitigate this, we
  propose an effective method of finding Pareto-optimal network architectures
  based on LLaMA2-7B using one-shot NAS. In particular, we fine-tune LLaMA2-7B
  only once and then apply genetic algorithm-based search to find smaller, less
  computationally complex network architectures. We show that, for certain
  standard benchmark tasks, the pre-trained LLaMA2-7B network is unnecessarily
  large and complex. More specifically, we demonstrate a 1.5x reduction in model
  size and 1.3x speedup in throughput for certain tasks with negligible drop in
  accuracy. In addition to finding smaller, higher-performing network
  architectures, our method does so more effectively and efficiently than certain
  pruning or sparsification techniques. Finally, we demonstrate how quantization
  is complementary to our method and that the size and complexity of the networks
  we find can be further decreased using quantization. We believe that our work
  provides a way to automatically create LLMs which can be used on less expensive
  and more readily available hardware platforms.
paper_authors._id: 66569cf93e1c895dd9189aca
paper_authors.name: Anthony Sarah
paper_authors._id: 66569cf93e1c895dd9189acb
paper_authors.name: Sharath Nittur Sridhar
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-05-29T03:11:54.086Z
paper_authors._id: 6604bc4a684646e2eb4b6628
paper_authors.avatarUrl: /avatars/7f32926b90de40d2dfcab53260c548e9.svg
paper_authors.fullname: Sharath 
paper_authors.type: user
paper_authors.user: nssharath
paper_authors._id: 66569cf93e1c895dd9189acc
paper_authors.name: Maciej Szankin
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-05-29T03:12:54.669Z
paper_authors._id: 63c83abc46421a2efe8160d0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63c83abc46421a2efe8160d0/Yy6IvAusEgxQ0qhCvB0Ka.jpeg
paper_authors.fullname: Mac Szankin
paper_authors.type: user
paper_authors.user: macsz
paper_authors._id: 66569cf93e1c895dd9189acd
paper_authors.name: Sairam Sundaresan
paper_title: LLaMA-NAS: Efficient Neural Architecture Search for Large Language
    Models
paper_summary: The abilities of modern large language models (LLMs) in solving natural
  language processing, complex reasoning, sentiment analysis and other tasks have
  been extraordinary which has prompted their extensive adoption. Unfortunately,
  these abilities come with very high memory and computational costs which
  precludes the use of LLMs on most hardware platforms. To mitigate this, we
  propose an effective method of finding Pareto-optimal network architectures
  based on LLaMA2-7B using one-shot NAS. In particular, we fine-tune LLaMA2-7B
  only once and then apply genetic algorithm-based search to find smaller, less
  computationally complex network architectures. We show that, for certain
  standard benchmark tasks, the pre-trained LLaMA2-7B network is unnecessarily
  large and complex. More specifically, we demonstrate a 1.5x reduction in model
  size and 1.3x speedup in throughput for certain tasks with negligible drop in
  accuracy. In addition to finding smaller, higher-performing network
  architectures, our method does so more effectively and efficiently than certain
  pruning or sparsification techniques. Finally, we demonstrate how quantization
  is complementary to our method and that the size and complexity of the networks
  we find can be further decreased using quantization. We believe that our work
  provides a way to automatically create LLMs which can be used on less expensive
  and more readily available hardware platforms.
paper_ai_keywords: LLaMA2-7B Pareto-optimal network architectures one-shot NAS fine-tune genetic algorithm-based search pre-trained LLaMA2-7B network model size throughput pruning sparsification quantization