In the context of BiomedGPT, x could refer to both
linguistic and visual tokens in the pretraining tasks,
including subwords, image codes, and location to-
kens, as we mentioned in Section 2.2. Specifically,
subwords are extracted by a BPE tokenizer, and we
mask 30% of the tokens of the subwords in input
in the MLM task as these medical words show rel-
atively high overlapping degrees. For the OD task,
location tokens are generated with Pix2Seq (Chen
et al., 2022a) conditioned on the observed pixel
inputs. We need data preprocessing for quantiz-
ing biomedical images using VQ-GAN (Esser et al.,
2021) because they are surrounded by trivial seman-
tics, e.g., black background and the unmet input
size. Therefore, we first remove the trivial back-
ground and crop the image to the bounding box of
the object of interest, then resize the cropped image
to be 256×256, and feed the center part with 128×128 resolution into the pre-trained VQ-GAN to generate
BiomedGPT