title: Emergence of Segmentation with Minimalistic White-Box Transformers
summary: Transformer-like models for vision tasks have recently proven effective for a
  wide range of downstream applications such as segmentation and detection.
  Previous works have shown that segmentation properties emerge in vision
  transformers (ViTs) trained using self-supervised methods such as DINO, but not
  in those trained on supervised classification tasks. In this study, we probe
  whether segmentation emerges in transformer-based models solely as a result of
  intricate self-supervised learning mechanisms, or if the same emergence can be
  achieved under much broader conditions through proper design of the model
  architecture. Through extensive experimental results, we demonstrate that when
  employing a white-box transformer-like architecture known as CRATE, whose
  design explicitly models and pursues low-dimensional structures in the data
  distribution, segmentation properties, at both the whole and parts levels,
  already emerge with a minimalistic supervised training recipe. Layer-wise
  finer-grained analysis reveals that the emergent properties strongly
  corroborate the designed mathematical functions of the white-box network. Our
  results suggest a path to design white-box foundation models that are
  simultaneously highly performant and mathematically fully interpretable. Code
  is at https://github.com/Ma-Lab-Berkeley/CRATE.
paper_authors._id: 64f1a1380af832a73d0f57f3
paper_authors.name: Yaodong Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-01T10:28:20.853Z
paper_authors._id: 6100e69a393be1b5c4c83867
paper_authors.avatarUrl: /avatars/1b87098cffb9c50345789808daea4f68.svg
paper_authors.fullname: Yaodong Yu
paper_authors.type: user
paper_authors.user: yaodongyu
paper_authors._id: 64f1a1380af832a73d0f57f4
paper_authors.name: Tianzhe Chu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-01T10:28:31.349Z
paper_authors._id: 641b83cba63c4e806236ae07
paper_authors.avatarUrl: /avatars/cf6e855fbe687005521865d9d501f2a1.svg
paper_authors.fullname: Tianzhe Chu
paper_authors.type: user
paper_authors.user: Tianzhe
paper_authors._id: 64f1a1380af832a73d0f57f5
paper_authors.name: Shengbang Tong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-01T10:31:11.419Z
paper_authors._id: 6374cbb7255276f3a22b4b35
paper_authors.avatarUrl: /avatars/7cf1bbb83447441e5fa2e1e4fcf7617b.svg
paper_authors.fullname: Peter Tong
paper_authors.type: user
paper_authors.user: tsbpp
paper_authors._id: 64f1a1380af832a73d0f57f6
paper_authors.name: Ziyang Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-01T10:32:36.292Z
paper_authors._id: 648fbaff4852c28c4cff7a3b
paper_authors.avatarUrl: /avatars/f987bb904580865676c8541839dd7615.svg
paper_authors.fullname: wuziyang
paper_authors.type: user
paper_authors.user: wzyohyes
paper_authors._id: 64f1a1380af832a73d0f57f7
paper_authors.name: Druv Pai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-01T10:29:14.590Z
paper_authors._id: 630194105ea4c617b21ccbf6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/630194105ea4c617b21ccbf6/mo8vVLwN84bzMxvWmMmF6.jpeg
paper_authors.fullname: Druv Pai
paper_authors.type: user
paper_authors.user: druvpai
paper_authors._id: 64f1a1380af832a73d0f57f8
paper_authors.name: Sam Buchanan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-01T10:29:27.608Z
paper_authors._id: 63069c1d41bf1fbadaf13b22
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661377504952-noauth.png
paper_authors.fullname: Sam Buchanan
paper_authors.type: user
paper_authors.user: sdbuch
paper_authors._id: 64f1a1380af832a73d0f57f9
paper_authors.name: Yi Ma
paper_title: Emergence of Segmentation with Minimalistic White-Box Transformers
paper_summary: Transformer-like models for vision tasks have recently proven effective for a
  wide range of downstream applications such as segmentation and detection.
  Previous works have shown that segmentation properties emerge in vision
  transformers (ViTs) trained using self-supervised methods such as DINO, but not
  in those trained on supervised classification tasks. In this study, we probe
  whether segmentation emerges in transformer-based models solely as a result of
  intricate self-supervised learning mechanisms, or if the same emergence can be
  achieved under much broader conditions through proper design of the model
  architecture. Through extensive experimental results, we demonstrate that when
  employing a white-box transformer-like architecture known as CRATE, whose
  design explicitly models and pursues low-dimensional structures in the data
  distribution, segmentation properties, at both the whole and parts levels,
  already emerge with a minimalistic supervised training recipe. Layer-wise
  finer-grained analysis reveals that the emergent properties strongly
  corroborate the designed mathematical functions of the white-box network. Our
  results suggest a path to design white-box foundation models that are
  simultaneously highly performant and mathematically fully interpretable. Code
  is at https://github.com/Ma-Lab-Berkeley/CRATE.
paper_ai_summary: Transformers designed with an emphasis on low-dimensional structures, such as CRATE, can develop segmentation properties even with minimal supervised training, challenging the notion that such properties require intricate self-supervised mechanisms.
paper_ai_keywords: vision transformers ViTs DINO CRATE white-box architecture self-supervised learning supervised training segmentation properties layer-wise analysis low-dimensional structures mathematical interpretability