Just a thought, @patrickvonplaten and @awsaf49 , in case we need pixel level classification, I guess ViT wont be that helpful, unless we are okay with a patch level segmentation. I saw that BigBird 1 implementation is available in JAX, maybe we could go with that since we could flatten the images to pixel level? The images seem to be 512 x 512. BigBird might need a lot of samples though since it needs to be trained from scratch…