LLaMA-NAS: Efficient Neural Architecture Search for Large Language
  Models
The abilities of modern large language models (LLMs) in solving natural
language processing, complex reasoning, sentiment analysis and other tasks have
been extraordinary which has prompted their extensive adoption. Unfortunately,
these abilities come with very high memory and computational costs which
precludes the use of LLMs on most hardware platforms. To mitigate this, we
propose an effective method of finding Pareto-optimal network architectures
based on LLaMA2-7B using one-shot NAS. In particular, we fine-tune LLaMA2-7B
only once and then apply genetic algorithm-based search to find smaller, less
computationally complex network architectures. We show that, for certain
standard benchmark tasks, the pre-trained LLaMA2-7B network is unnecessarily
large and complex. More specifically, we demonstrate a 1.5x reduction in model
size and 1.3x speedup in throughput for certain tasks with negligible drop in
accuracy. In addition to finding smaller, higher-performing network
architectures, our method does so more effectively and efficiently than certain
pruning or sparsification techniques. Finally, we demonstrate how quantization
is complementary to our method and that the size and complexity of the networks
we find can be further decreased using quantization. We believe that our work
provides a way to automatically create LLMs which can be used on less expensive
and more readily available hardware platforms.
66569cf93e1c895dd9189aca
Anthony Sarah
66569cf93e1c895dd9189acb
Sharath Nittur Sridhar
extracted_pending
2024-05-29T03:11:54.086Z
6604bc4a684646e2eb4b6628
/avatars/7f32926b90de40d2dfcab53260c548e9.svg
Sharath 
user
nssharath
66569cf93e1c895dd9189acc
Maciej Szankin
extracted_confirmed
2024-05-29T03:12:54.669Z
63c83abc46421a2efe8160d0
https://cdn-avatars.huggingface.co/v1/production/uploads/63c83abc46421a2efe8160d0/Yy6IvAusEgxQ0qhCvB0Ka.jpeg
Mac Szankin
user
macsz
66569cf93e1c895dd9189acd
Sairam Sundaresan
LLaMA-NAS: Efficient Neural Architecture Search for Large Language
  Models
The abilities of modern large language models (LLMs) in solving natural
language processing, complex reasoning, sentiment analysis and other tasks have
been extraordinary which has prompted their extensive adoption. Unfortunately,
these abilities come with very high memory and computational costs which
precludes the use of LLMs on most hardware platforms. To mitigate this, we
propose an effective method of finding Pareto-optimal network architectures
based on LLaMA2-7B using one-shot NAS. In particular, we fine-tune LLaMA2-7B
only once and then apply genetic algorithm-based search to find smaller, less
computationally complex network architectures. We show that, for certain
standard benchmark tasks, the pre-trained LLaMA2-7B network is unnecessarily
large and complex. More specifically, we demonstrate a 1.5x reduction in model
size and 1.3x speedup in throughput for certain tasks with negligible drop in
accuracy. In addition to finding smaller, higher-performing network
architectures, our method does so more effectively and efficiently than certain
pruning or sparsification techniques. Finally, we demonstrate how quantization
is complementary to our method and that the size and complexity of the networks
we find can be further decreased using quantization. We believe that our work
provides a way to automatically create LLMs which can be used on less expensive
and more readily available hardware platforms.
A method using one-shot NAS and genetic algorithms identifies smaller, more efficient LLaMA2-7B architectures with reduced size and increased throughput for various tasks, complemented by quantization.
one-shot NAS genetic algorithm LLaMA2-7B Pareto-optimal network architectures model size reduction throughput improvement pruning sparsification quantization