that the model had mistakenly interpreted the VQA task as an image generation task. Although both
BiomedGPTMedisum/Small and OFABase generate text-only answer “No", it still does not match the open-
ended ground truth of “Head”. Figure 5 displays the open- and closed-ended accuracies in transfer learning
on VQA-RAD data. We observed that models exhibit catastrophic performance on open-ended questions.
Note that there are 251 out of 451 closed-ended QA pairs (either “Yes” or “No”) in the VQA-RAD test set,
with the remainder being open questions.
These observations shed light on the instruction-sensitivity challenge that arises in instruction-guided pre-
training when building a unified and generalist biomedical model. To delve deeper into this issue, we present
a case study in Appendix A. Additionally, our findings from the intra- and inter-distribution transfer learning
experiments suggest that data diversity and scale are crucial factors in enhancing the generalist intelligence
BiomedGPT