title: Energy Efficient Protein Language Models: Leveraging Small Language
    Models with LoRA for Controllable Protein Generation
summary: Large language models (LLMs) have demonstrated significant success in natural
  language processing (NLP) tasks and have shown promising results in other
  domains such as protein sequence generation. However, there remain salient
  differences between LLMs used for NLP, which effectively handle multiple tasks
  and are available in small sizes, and protein language models that are often
  specialized for specific tasks and only exist in larger sizes. In this work, we
  introduce two small protein language models, based on Llama-3-8B and
  Phi-3-mini, that are capable of both uncontrollable and controllable protein
  generation. For the uncontrollable generation task, our best model achieves an
  average pLDDT score of 69.75, demonstrating robust performance in generating
  viable protein structures. For the controllable generation task, in which the
  model generates proteins according to properties specified in the prompt, we
  achieve a remarkable average TM-Score of 0.84, indicating high structural
  similarity to target proteins. We chose 10 properties, including six classes of
  enzymes, to extend the capabilities of prior protein language models. Our
  approach utilizes the Low-Rank Adaptor (LoRA) technique, reducing trainable
  parameters to just 4% of the original model size, lowering computational
  requirements. By using a subset of the UniRef50 dataset and small models, we
  reduced the overall training time by 70% without compromising performance.
  Notably, Phi-3-mini reduced trainable parameters by 60%, decreasing training
  cost by 30% compared to Llama 3. Consequently, Phi-3 achieved a comparable
  TM-Score of 0.81, demonstrating that smaller models can match the performance
  of larger ones, like Llama 3. We also demonstrate the deployment of our models
  on the energy efficient ET-SoC-1 chip, significantly improving the TPS/W by a
  factor of 3.
paper_authors._id: 6732d42c22787fbe3c623846
paper_authors.name: Aayush Shah
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-11-19T21:27:31.655Z
paper_authors._id: 664e38c072b3c9f674067a68
paper_authors.avatarUrl: /avatars/ffa0db57c4b9088c891da1385c4d98b1.svg
paper_authors.fullname: Aayush Shah
paper_authors.type: user
paper_authors.user: aayush-shah
paper_authors._id: 6732d42c22787fbe3c623847
paper_authors.name: Shankar Jayaratnam
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-11-12T04:10:10.707Z
paper_authors._id: 66260312908e16c6fbf612c9
paper_authors.avatarUrl: /avatars/ea56148b40b3c0228e145080561636bb.svg
paper_authors.fullname: Shankar Jayaratnam
paper_authors.type: user
paper_authors.user: ShankarJay
paper_title: Energy Efficient Protein Language Models: Leveraging Small Language
    Models with LoRA for Controllable Protein Generation
paper_summary: Large language models (LLMs) have demonstrated significant success in natural
  language processing (NLP) tasks and have shown promising results in other
  domains such as protein sequence generation. However, there remain salient
  differences between LLMs used for NLP, which effectively handle multiple tasks
  and are available in small sizes, and protein language models that are often
  specialized for specific tasks and only exist in larger sizes. In this work, we
  introduce two small protein language models, based on Llama-3-8B and
  Phi-3-mini, that are capable of both uncontrollable and controllable protein
  generation. For the uncontrollable generation task, our best model achieves an
  average pLDDT score of 69.75, demonstrating robust performance in generating
  viable protein structures. For the controllable generation task, in which the
  model generates proteins according to properties specified in the prompt, we
  achieve a remarkable average TM-Score of 0.84, indicating high structural
  similarity to target proteins. We chose 10 properties, including six classes of
  enzymes, to extend the capabilities of prior protein language models. Our
  approach utilizes the Low-Rank Adaptor (LoRA) technique, reducing trainable
  parameters to just 4% of the original model size, lowering computational
  requirements. By using a subset of the UniRef50 dataset and small models, we
  reduced the overall training time by 70% without compromising performance.
  Notably, Phi-3-mini reduced trainable parameters by 60%, decreasing training
  cost by 30% compared to Llama 3. Consequently, Phi-3 achieved a comparable
  TM-Score of 0.81, demonstrating that smaller models can match the performance
  of larger ones, like Llama 3. We also demonstrate the deployment of our models
  on the energy efficient ET-SoC-1 chip, significantly improving the TPS/W by a
  factor of 3.
paper_ai_keywords: Llama-3-8B Phi-3-mini protein language models pLDDT score TM-Score enzyme classes Low-Rank Adaptor (LoRA) UniRef50 dataset ET-SoC-1 chip TPS/W