FGBERT: Function-Driven Pre-trained Gene Language Model for Metagenomics
<|reference_start|>FGBERT: Function-Driven Pre-trained Gene Language Model for Metagenomics: Metagenomic data, comprising mixed multi-species genomes, are prevalent in diverse environments like oceans and soils, significantly impacting human health and ecological functions. However, current research relies on K-mer representations, limiting the capture of structurally relevant gene contexts. To address these limitations and further our understanding of complex relationships between metagenomic sequences and their functions, we introduce a protein-based gene representation as a context-aware and structure-relevant tokenizer. Our approach includes Masked Gene Modeling (MGM) for gene group-level pre-training, providing insights into inter-gene contextual information, and Triple Enhanced Metagenomic Contrastive Learning (TEM-CL) for gene-level pre-training to model gene sequence-function relationships. MGM and TEM-CL constitute our novel metagenomic language model {\NAME}, pre-trained on 100 million metagenomic sequences. We demonstrate the superiority of our proposed {\NAME} on eight datasets.<|reference_end|>
@article{duan2024fgbert:,
    title={FGBERT: Function-Driven Pre-trained Gene Language Model for Metagenomics},
    author={ChenRui Duan, Zelin Zang, Yongjie Xu, Hang He, Zihan Liu, Zijia Song,
  Ju-Sheng Zheng, Stan Z. Li},
    journal={arXiv preprint arXiv:2402.16901},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.16901},
    primaryClass={q-bio.GN cs.AI cs.LG}
}