A Spitting Image: Modular Superpixel Tokenization in Vision Transformers
<|reference_start|>A Spitting Image: Modular Superpixel Tokenization in Vision Transformers: Vision Transformer (ViT) architectures traditionally employ a grid-based approach to tokenization independent of the semantic content of an image. We propose a modular superpixel tokenization strategy which decouples tokenization and feature extraction; a shift from contemporary approaches where these are treated as an undifferentiated whole. Using on-line content-aware tokenization and scale- and shape-invariant positional embeddings, we perform experiments and ablations that contrast our approach with patch-based tokenization and randomized partitions as baselines. We show that our method significantly improves the faithfulness of attributions, gives pixel-level granularity on zero-shot unsupervised dense prediction tasks, while maintaining predictive performance in classification tasks. Our approach provides a modular tokenization framework commensurable with standard architectures, extending the space of ViTs to a larger class of semantically-rich models.<|reference_end|>
@article{aasan2024a,
    title={A Spitting Image: Modular Superpixel Tokenization in Vision Transformers},
    author={Marius Aasan, Odd Kolbj{o}rnsen, Anne Schistad Solberg, Ad'in
  Ramirez Rivera},
    journal={arXiv preprint arXiv:2408.07680},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.07680},
    primaryClass={cs.CV cs.AI cs.LG}
}