the optimal sequence for training with diverse inputs. Another potential avenue is to align BiomedGPT
with human intent via reinforcement learning from human or AI feedback (RLF) (Ouyang et al., 2022; Bai
et al., 2022; Zhou et al., 2023), a strategy employed by the latest dialogue agents such as ChatGPT6 and
Claude7. However, creating specific biomedical RLF datasets would be expensive, given the extensive need
for domain experts.
A further significant limitation arises from two specific inputs for text-only downstream tasks: the consider-
able difference between clinical notes and general-domain & biomedical text (Lehman et al., 2023; Gu et al.,
2021), and the presence of vision-only tasks, which can impede the model’s pattern extraction from the pure
text during pretraining, as highlighted in our ablation study in Section 3.5. Generating a representative
vocabulary from all domains and increasing the ratio of text inputs during pretraining may help address
BiomedGPT