title: Your ViT is Secretly an Image Segmentation Model
summary: Vision Transformers (ViTs) have shown remarkable performance and scalability
  across various computer vision tasks. To apply single-scale ViTs to image
  segmentation, existing methods adopt a convolutional adapter to generate
  multi-scale features, a pixel decoder to fuse these features, and a Transformer
  decoder that uses the fused features to make predictions. In this paper, we
  show that the inductive biases introduced by these task-specific components can
  instead be learned by the ViT itself, given sufficiently large models and
  extensive pre-training. Based on these findings, we introduce the Encoder-only
  Mask Transformer (EoMT), which repurposes the plain ViT architecture to conduct
  image segmentation. With large-scale models and pre-training, EoMT obtains a
  segmentation accuracy similar to state-of-the-art models that use task-specific
  components. At the same time, EoMT is significantly faster than these methods
  due to its architectural simplicity, e.g., up to 4x faster with ViT-L. Across a
  range of model sizes, EoMT demonstrates an optimal balance between segmentation
  accuracy and prediction speed, suggesting that compute resources are better
  spent on scaling the ViT itself rather than adding architectural complexity.
  Code: https://www.tue-mps.org/eomt/.
paper_authors._id: 67ea6d5a2d95c10a0d921c6b
paper_authors.name: Tommie Kerssies
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-31T12:57:35.964Z
paper_authors._id: 6368212544e19ccad212bbf2
paper_authors.avatarUrl: /avatars/3ab0d1c6273a95f16b7b82c72db4a5e2.svg
paper_authors.fullname: Tommie Kerssies
paper_authors.type: user
paper_authors.user: tommiekerssies
paper_authors._id: 67ea6d5a2d95c10a0d921c6c
paper_authors.name: Niccolò Cavagnero
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-31T12:57:32.351Z
paper_authors._id: 67ea650e28179c61bef30406
paper_authors.avatarUrl: /avatars/cad077d3185386fc2a0b49fbf724ca5e.svg
paper_authors.fullname: Niccolò Cavagnero
paper_authors.type: user
paper_authors.user: neikos00
paper_authors._id: 67ea6d5a2d95c10a0d921c6d
paper_authors.name: Alexander Hermans
paper_authors._id: 67ea6d5a2d95c10a0d921c6e
paper_authors.name: Narges Norouzi
paper_authors._id: 67ea6d5a2d95c10a0d921c6f
paper_authors.name: Giuseppe Averta
paper_authors._id: 67ea6d5a2d95c10a0d921c70
paper_authors.name: Bastian Leibe
paper_authors._id: 67ea6d5a2d95c10a0d921c71
paper_authors.name: Gijs Dubbelman
paper_authors._id: 67ea6d5a2d95c10a0d921c72
paper_authors.name: Daan de Geus
paper_title: Your ViT is Secretly an Image Segmentation Model
paper_summary: Vision Transformers (ViTs) have shown remarkable performance and scalability
  across various computer vision tasks. To apply single-scale ViTs to image
  segmentation, existing methods adopt a convolutional adapter to generate
  multi-scale features, a pixel decoder to fuse these features, and a Transformer
  decoder that uses the fused features to make predictions. In this paper, we
  show that the inductive biases introduced by these task-specific components can
  instead be learned by the ViT itself, given sufficiently large models and
  extensive pre-training. Based on these findings, we introduce the Encoder-only
  Mask Transformer (EoMT), which repurposes the plain ViT architecture to conduct
  image segmentation. With large-scale models and pre-training, EoMT obtains a
  segmentation accuracy similar to state-of-the-art models that use task-specific
  components. At the same time, EoMT is significantly faster than these methods
  due to its architectural simplicity, e.g., up to 4x faster with ViT-L. Across a
  range of model sizes, EoMT demonstrates an optimal balance between segmentation
  accuracy and prediction speed, suggesting that compute resources are better
  spent on scaling the ViT itself rather than adding architectural complexity.
  Code: https://www.tue-mps.org/eomt/.
paper_ai_summary: The Encoder-only Mask Transformer (EoMT) achieves state-of-the-art image segmentation accuracy by learning task-specific components through large-scale pre-training, while maintaining superior prediction speed compared to models with additional architectural complexity.
paper_ai_keywords: Vision Transformers ViTs image segmentation multi-scale features pixel decoder Transformer decoder inductive biases Encoder-only Mask Transformer EoMT large-scale pre-training prediction speed