Your ViT is Secretly an Image Segmentation Model
Vision Transformers (ViTs) have shown remarkable performance and scalability
across various computer vision tasks. To apply single-scale ViTs to image
segmentation, existing methods adopt a convolutional adapter to generate
multi-scale features, a pixel decoder to fuse these features, and a Transformer
decoder that uses the fused features to make predictions. In this paper, we
show that the inductive biases introduced by these task-specific components can
instead be learned by the ViT itself, given sufficiently large models and
extensive pre-training. Based on these findings, we introduce the Encoder-only
Mask Transformer (EoMT), which repurposes the plain ViT architecture to conduct
image segmentation. With large-scale models and pre-training, EoMT obtains a
segmentation accuracy similar to state-of-the-art models that use task-specific
components. At the same time, EoMT is significantly faster than these methods
due to its architectural simplicity, e.g., up to 4x faster with ViT-L. Across a
range of model sizes, EoMT demonstrates an optimal balance between segmentation
accuracy and prediction speed, suggesting that compute resources are better
spent on scaling the ViT itself rather than adding architectural complexity.
Code: https://www.tue-mps.org/eomt/.
67ea6d5a2d95c10a0d921c6b
Tommie Kerssies
claimed_verified
2025-03-31T12:57:35.964Z
6368212544e19ccad212bbf2
/avatars/3ab0d1c6273a95f16b7b82c72db4a5e2.svg
Tommie Kerssies
user
tommiekerssies
67ea6d5a2d95c10a0d921c6c
Niccolò Cavagnero
claimed_verified
2025-03-31T12:57:32.351Z
67ea650e28179c61bef30406
/avatars/bb142e1a8abf29be7d04bcbdf4b2dab5.svg
Niccolò Cavagnero
user
neikos00
67ea6d5a2d95c10a0d921c6d
Alexander Hermans
67ea6d5a2d95c10a0d921c6e
Narges Norouzi
67ea6d5a2d95c10a0d921c6f
Giuseppe Averta
67ea6d5a2d95c10a0d921c70
Bastian Leibe
67ea6d5a2d95c10a0d921c71
Gijs Dubbelman
67ea6d5a2d95c10a0d921c72
Daan de Geus
claimed_verified
2025-07-01T10:15:26.571Z
6392408ae836c1426b776472
/avatars/380b66f142197572e4d8516046c82508.svg
Daan de Geus
user
ddegeus
Your ViT is Secretly an Image Segmentation Model
Vision Transformers (ViTs) have shown remarkable performance and scalability
across various computer vision tasks. To apply single-scale ViTs to image
segmentation, existing methods adopt a convolutional adapter to generate
multi-scale features, a pixel decoder to fuse these features, and a Transformer
decoder that uses the fused features to make predictions. In this paper, we
show that the inductive biases introduced by these task-specific components can
instead be learned by the ViT itself, given sufficiently large models and
extensive pre-training. Based on these findings, we introduce the Encoder-only
Mask Transformer (EoMT), which repurposes the plain ViT architecture to conduct
image segmentation. With large-scale models and pre-training, EoMT obtains a
segmentation accuracy similar to state-of-the-art models that use task-specific
components. At the same time, EoMT is significantly faster than these methods
due to its architectural simplicity, e.g., up to 4x faster with ViT-L. Across a
range of model sizes, EoMT demonstrates an optimal balance between segmentation
accuracy and prediction speed, suggesting that compute resources are better
spent on scaling the ViT itself rather than adding architectural complexity.
Code: https://www.tue-mps.org/eomt/.
The Encoder-only Mask Transformer (EoMT) achieves state-of-the-art image segmentation accuracy by learning task-specific components through large-scale pre-training, while maintaining superior prediction speed compared to models with additional architectural complexity.
Vision Transformers ViTs image segmentation multi-scale features pixel decoder Transformer decoder inductive biases Encoder-only Mask Transformer EoMT large-scale pre-training prediction speed