In the rapidly evolving landscape of artificial intelligence (AI), transformer-based foundation models (Vaswani
et al., 2017; Dosovitskiy et al., 2020; Bommasani et al., 2021; Zhou et al., 2023) have emerged as a powerful
tool for solving a wide range of biomedical challenges, e.g., radiograph analysis (Park et al., 2022; Zhou
et al., 2022), biomedical text generation (Luo et al., 2022), disease prediction (Rasmy et al., 2021), and
cell type annotation (Yang et al., 2022a). The prevailing paradigm for biomedical foundation models is the
pretraining then fine-tuning. Specifically, a model is first pre-trained on a large-scale dataset and then fine-
tuned on downstream datasets, facilitating knowledge transfer from the source domain to the target domain
(Niu et al., 2020). Instead of supervised pretraining, which is constrained by the availability of labeled data,
self-supervised approaches that can learn from vast amounts of data without explicit human labeling have
BiomedGPT