Emergence of Segmentation with Minimalistic White-Box Transformers
Transformer-like models for vision tasks have recently proven effective for a
wide range of downstream applications such as segmentation and detection.
Previous works have shown that segmentation properties emerge in vision
transformers (ViTs) trained using self-supervised methods such as DINO, but not
in those trained on supervised classification tasks. In this study, we probe
whether segmentation emerges in transformer-based models solely as a result of
intricate self-supervised learning mechanisms, or if the same emergence can be
achieved under much broader conditions through proper design of the model
architecture. Through extensive experimental results, we demonstrate that when
employing a white-box transformer-like architecture known as CRATE, whose
design explicitly models and pursues low-dimensional structures in the data
distribution, segmentation properties, at both the whole and parts levels,
already emerge with a minimalistic supervised training recipe. Layer-wise
finer-grained analysis reveals that the emergent properties strongly
corroborate the designed mathematical functions of the white-box network. Our
results suggest a path to design white-box foundation models that are
simultaneously highly performant and mathematically fully interpretable. Code
is at https://github.com/Ma-Lab-Berkeley/CRATE.
64f1a1380af832a73d0f57f3
Yaodong Yu
admin_assigned
2023-09-01T10:28:20.853Z
6100e69a393be1b5c4c83867
/avatars/1b87098cffb9c50345789808daea4f68.svg
Yaodong Yu
user
yaodongyu
64f1a1380af832a73d0f57f4
Tianzhe Chu
admin_assigned
2023-09-01T10:28:31.349Z
641b83cba63c4e806236ae07
/avatars/cf6e855fbe687005521865d9d501f2a1.svg
Tianzhe Chu
user
Tianzhe
64f1a1380af832a73d0f57f5
Shengbang Tong
admin_assigned
2023-09-01T10:31:11.419Z
6374cbb7255276f3a22b4b35
/avatars/7cf1bbb83447441e5fa2e1e4fcf7617b.svg
Peter Tong
user
tsbpp
64f1a1380af832a73d0f57f6
Ziyang Wu
admin_assigned
2023-09-01T10:32:36.292Z
648fbaff4852c28c4cff7a3b
/avatars/f987bb904580865676c8541839dd7615.svg
wuziyang
user
wzyohyes
64f1a1380af832a73d0f57f7
Druv Pai
admin_assigned
2023-09-01T10:29:14.590Z
630194105ea4c617b21ccbf6
https://cdn-avatars.huggingface.co/v1/production/uploads/630194105ea4c617b21ccbf6/mo8vVLwN84bzMxvWmMmF6.jpeg
Druv Pai
user
druvpai
64f1a1380af832a73d0f57f8
Sam Buchanan
admin_assigned
2023-09-01T10:29:27.608Z
63069c1d41bf1fbadaf13b22
https://cdn-avatars.huggingface.co/v1/production/uploads/1661377504952-noauth.png
Sam Buchanan
user
sdbuch
64f1a1380af832a73d0f57f9
Yi Ma
Emergence of Segmentation with Minimalistic White-Box Transformers
Transformer-like models for vision tasks have recently proven effective for a
wide range of downstream applications such as segmentation and detection.
Previous works have shown that segmentation properties emerge in vision
transformers (ViTs) trained using self-supervised methods such as DINO, but not
in those trained on supervised classification tasks. In this study, we probe
whether segmentation emerges in transformer-based models solely as a result of
intricate self-supervised learning mechanisms, or if the same emergence can be
achieved under much broader conditions through proper design of the model
architecture. Through extensive experimental results, we demonstrate that when
employing a white-box transformer-like architecture known as CRATE, whose
design explicitly models and pursues low-dimensional structures in the data
distribution, segmentation properties, at both the whole and parts levels,
already emerge with a minimalistic supervised training recipe. Layer-wise
finer-grained analysis reveals that the emergent properties strongly
corroborate the designed mathematical functions of the white-box network. Our
results suggest a path to design white-box foundation models that are
simultaneously highly performant and mathematically fully interpretable. Code
is at https://github.com/Ma-Lab-Berkeley/CRATE.
Transformers designed with an emphasis on low-dimensional structures, such as CRATE, can develop segmentation properties even with minimal supervised training, challenging the notion that such properties require intricate self-supervised mechanisms.
vision transformers ViTs DINO CRATE white-box architecture self-supervised learning supervised training segmentation properties layer-wise analysis low-dimensional structures mathematical interpretability