title: Ankh3: Multi-Task Pretraining with Sequence Denoising and Completion
    Enhances Protein Representations
summary: Protein language models (PLMs) have emerged as powerful tools to detect
  complex patterns of protein sequences. However, the capability of PLMs to fully
  capture information on protein sequences might be limited by focusing on single
  pre-training tasks. Although adding data modalities or supervised objectives
  can improve the performance of PLMs, pre-training often remains focused on
  denoising corrupted sequences. To push the boundaries of PLMs, our research
  investigated a multi-task pre-training strategy. We developed Ankh3, a model
  jointly optimized on two objectives: masked language modeling with multiple
  masking probabilities and protein sequence completion relying only on protein
  sequences as input. This multi-task pre-training demonstrated that PLMs can
  learn richer and more generalizable representations solely from protein
  sequences. The results demonstrated improved performance in downstream tasks,
  such as secondary structure prediction, fluorescence, GB1 fitness, and contact
  prediction. The integration of multiple tasks gave the model a more
  comprehensive understanding of protein properties, leading to more robust and
  accurate predictions.
paper_authors._id: 6836f9a6d44ecc28c70cc038
paper_authors.name: Hazem Alsamkary
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T12:12:23.522Z
paper_authors._id: 61faa58a11deaebce017da4a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1670851610124-61faa58a11deaebce017da4a.jpeg
paper_authors.fullname: Hazem Essam
paper_authors.type: user
paper_authors.user: hazemessam
paper_authors._id: 6836f9a6d44ecc28c70cc039
paper_authors.name: Mohamed Elshaffei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-28T12:27:38.964Z
paper_authors._id: 63959596401bd73e467e1fc2
paper_authors.avatarUrl: /avatars/963d073e9d5ee414d5c185faf9d8d4a6.svg
paper_authors.fullname: Mohamed Elshaffei
paper_authors.type: user
paper_authors.user: shaffei
paper_authors._id: 6836f9a6d44ecc28c70cc03a
paper_authors.name: Mohamed Elkerdawy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-28T12:27:45.103Z
paper_authors._id: 62f96db84588fe31f43aceb9
paper_authors.avatarUrl: /avatars/7577f071b5a030f04f6ab0df46e5e246.svg
paper_authors.fullname: Mohamed Elkerdawy
paper_authors.type: user
paper_authors.user: melkerdawy
paper_authors._id: 6836f9a6d44ecc28c70cc03b
paper_authors.name: Ahmed Elnaggar
paper_title: Ankh3: Multi-Task Pretraining with Sequence Denoising and Completion
    Enhances Protein Representations
paper_summary: Protein language models (PLMs) have emerged as powerful tools to detect
  complex patterns of protein sequences. However, the capability of PLMs to fully
  capture information on protein sequences might be limited by focusing on single
  pre-training tasks. Although adding data modalities or supervised objectives
  can improve the performance of PLMs, pre-training often remains focused on
  denoising corrupted sequences. To push the boundaries of PLMs, our research
  investigated a multi-task pre-training strategy. We developed Ankh3, a model
  jointly optimized on two objectives: masked language modeling with multiple
  masking probabilities and protein sequence completion relying only on protein
  sequences as input. This multi-task pre-training demonstrated that PLMs can
  learn richer and more generalizable representations solely from protein
  sequences. The results demonstrated improved performance in downstream tasks,
  such as secondary structure prediction, fluorescence, GB1 fitness, and contact
  prediction. The integration of multiple tasks gave the model a more
  comprehensive understanding of protein properties, leading to more robust and
  accurate predictions.
paper_ai_summary: A multi-task pre-training strategy for protein language models improves their performance on downstream protein prediction tasks by learning richer representations from sequence data alone.
paper_ai_keywords: protein language models pre-training masked language modeling protein sequence completion secondary structure prediction fluorescence fitness contact prediction