Energy Efficient Protein Language Models: Leveraging Small Language
  Models with LoRA for Controllable Protein Generation
Large language models (LLMs) have demonstrated significant success in natural
language processing (NLP) tasks and have shown promising results in other
domains such as protein sequence generation. However, there remain salient
differences between LLMs used for NLP, which effectively handle multiple tasks
and are available in small sizes, and protein language models that are often
specialized for specific tasks and only exist in larger sizes. In this work, we
introduce two small protein language models, based on Llama-3-8B and
Phi-3-mini, that are capable of both uncontrollable and controllable protein
generation. For the uncontrollable generation task, our best model achieves an
average pLDDT score of 69.75, demonstrating robust performance in generating
viable protein structures. For the controllable generation task, in which the
model generates proteins according to properties specified in the prompt, we
achieve a remarkable average TM-Score of 0.84, indicating high structural
similarity to target proteins. We chose 10 properties, including six classes of
enzymes, to extend the capabilities of prior protein language models. Our
approach utilizes the Low-Rank Adaptor (LoRA) technique, reducing trainable
parameters to just 4% of the original model size, lowering computational
requirements. By using a subset of the UniRef50 dataset and small models, we
reduced the overall training time by 70% without compromising performance.
Notably, Phi-3-mini reduced trainable parameters by 60%, decreasing training
cost by 30% compared to Llama 3. Consequently, Phi-3 achieved a comparable
TM-Score of 0.81, demonstrating that smaller models can match the performance
of larger ones, like Llama 3. We also demonstrate the deployment of our models
on the energy efficient ET-SoC-1 chip, significantly improving the TPS/W by a
factor of 3.
6732d42c22787fbe3c623846
Aayush Shah
extracted_confirmed
2024-11-19T21:27:31.655Z
664e38c072b3c9f674067a68
/avatars/4de95dad7837c36e3ec3d101510e8568.svg
Aayush Shah
user
aayush-shah
6732d42c22787fbe3c623847
Shankar Jayaratnam
extracted_confirmed
2024-11-12T04:10:10.707Z
66260312908e16c6fbf612c9
/avatars/ea56148b40b3c0228e145080561636bb.svg
Shankar Jayaratnam
user
ShankarJay
Energy Efficient Protein Language Models: Leveraging Small Language
  Models with LoRA for Controllable Protein Generation
Large language models (LLMs) have demonstrated significant success in natural
language processing (NLP) tasks and have shown promising results in other
domains such as protein sequence generation. However, there remain salient
differences between LLMs used for NLP, which effectively handle multiple tasks
and are available in small sizes, and protein language models that are often
specialized for specific tasks and only exist in larger sizes. In this work, we
introduce two small protein language models, based on Llama-3-8B and
Phi-3-mini, that are capable of both uncontrollable and controllable protein
generation. For the uncontrollable generation task, our best model achieves an
average pLDDT score of 69.75, demonstrating robust performance in generating
viable protein structures. For the controllable generation task, in which the
model generates proteins according to properties specified in the prompt, we
achieve a remarkable average TM-Score of 0.84, indicating high structural
similarity to target proteins. We chose 10 properties, including six classes of
enzymes, to extend the capabilities of prior protein language models. Our
approach utilizes the Low-Rank Adaptor (LoRA) technique, reducing trainable
parameters to just 4% of the original model size, lowering computational
requirements. By using a subset of the UniRef50 dataset and small models, we
reduced the overall training time by 70% without compromising performance.
Notably, Phi-3-mini reduced trainable parameters by 60%, decreasing training
cost by 30% compared to Llama 3. Consequently, Phi-3 achieved a comparable
TM-Score of 0.81, demonstrating that smaller models can match the performance
of larger ones, like Llama 3. We also demonstrate the deployment of our models
on the energy efficient ET-SoC-1 chip, significantly improving the TPS/W by a
factor of 3.
Two small protein language models, based on Llama-3-8B and Phi-3-mini, achieve comparable performance to larger models in protein generation tasks using parameter-efficient adaptation techniques.
LLMs NLP protein sequence generation protein language models uncontrollable generation controllable generation pLDDT score TM-Score Low-Rank Adaptor (LoRA) UniRef50 dataset ET-SoC-1 chip TPS/W