title: Rethinking Patch Dependence for Masked Autoencoders
summary: In this work, we re-examine inter-patch dependencies in the decoding
  mechanism of masked autoencoders (MAE). We decompose this decoding mechanism
  for masked patch reconstruction in MAE into self-attention and cross-attention.
  Our investigations suggest that self-attention between mask patches is not
  essential for learning good representations. To this end, we propose a novel
  pretraining framework: Cross-Attention Masked Autoencoders (CrossMAE).
  CrossMAE's decoder leverages only cross-attention between masked and visible
  tokens, with no degradation in downstream performance. This design also enables
  decoding only a small subset of mask tokens, boosting efficiency. Furthermore,
  each decoder block can now leverage different encoder features, resulting in
  improved representation learning. CrossMAE matches MAE in performance with 2.5
  to 3.7times less decoding compute. It also surpasses MAE on ImageNet
  classification and COCO instance segmentation under the same compute. Code and
  models: https://crossmae.github.io
paper_authors._id: 65b31a603a41095a56ae0fc5
paper_authors.name: Letian Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T08:22:58.402Z
paper_authors._id: 64908d6b0c18343a09361b93
paper_authors.avatarUrl: /avatars/2fa44344fe760f68f19de775b76f45b8.svg
paper_authors.fullname: Max (Letian) Fu
paper_authors.type: user
paper_authors.user: mlfu7
paper_authors._id: 65b31a603a41095a56ae0fc6
paper_authors.name: Long Lian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T08:25:25.404Z
paper_authors._id: 63797c273f575acc2f6893c0
paper_authors.avatarUrl: /avatars/32d7a6a8881c8c4d80a097b732ed24b6.svg
paper_authors.fullname: Long(Tony) Lian
paper_authors.type: user
paper_authors.user: longlian
paper_authors._id: 65b31a603a41095a56ae0fc7
paper_authors.name: Renhao Wang
paper_authors._id: 65b31a603a41095a56ae0fc8
paper_authors.name: Baifeng Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T08:26:01.815Z
paper_authors._id: 649004218f7cbbc94c782db6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/AdgLVfAIpWlug4jXTaEK-.jpeg
paper_authors.fullname: Baifeng Shi
paper_authors.type: user
paper_authors.user: bfshi
paper_authors._id: 65b31a603a41095a56ae0fc9
paper_authors.name: Xudong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T08:26:35.984Z
paper_authors._id: 63d6f76f44f1d8fbe58fbd9d
paper_authors.avatarUrl: /avatars/184f248a090b543e576a97b7f46ab3ef.svg
paper_authors.fullname: Xudong Wang
paper_authors.type: user
paper_authors.user: xudongw
paper_authors._id: 65b31a603a41095a56ae0fca
paper_authors.name: Adam Yala
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T08:26:55.544Z
paper_authors._id: 6333a9195a032dcd095dda13
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1664329996201-noauth.jpeg
paper_authors.fullname: Adam Yala
paper_authors.type: user
paper_authors.user: yala
paper_authors._id: 65b31a603a41095a56ae0fcb
paper_authors.name: Trevor Darrell
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T08:27:03.308Z
paper_authors._id: 64cbdf02f103036e23d1c7f3
paper_authors.avatarUrl: /avatars/496069463900dea20929b57381182d39.svg
paper_authors.fullname: Trevor Darrell
paper_authors.type: user
paper_authors.user: trevordarrell
paper_authors._id: 65b31a603a41095a56ae0fcc
paper_authors.name: Alexei A. Efros
paper_authors._id: 65b31a603a41095a56ae0fcd
paper_authors.name: Ken Goldberg
paper_title: Rethinking Patch Dependence for Masked Autoencoders
paper_summary: In this work, we re-examine inter-patch dependencies in the decoding
  mechanism of masked autoencoders (MAE). We decompose this decoding mechanism
  for masked patch reconstruction in MAE into self-attention and cross-attention.
  Our investigations suggest that self-attention between mask patches is not
  essential for learning good representations. To this end, we propose a novel
  pretraining framework: Cross-Attention Masked Autoencoders (CrossMAE).
  CrossMAE's decoder leverages only cross-attention between masked and visible
  tokens, with no degradation in downstream performance. This design also enables
  decoding only a small subset of mask tokens, boosting efficiency. Furthermore,
  each decoder block can now leverage different encoder features, resulting in
  improved representation learning. CrossMAE matches MAE in performance with 2.5
  to 3.7times less decoding compute. It also surpasses MAE on ImageNet
  classification and COCO instance segmentation under the same compute. Code and
  models: https://crossmae.github.io
paper_ai_summary: By using only cross-attention for masked patch reconstruction, CrossMAE achieves comparable and sometimes superior performance to MAE with significantly reduced computational cost.
paper_ai_keywords: masked autoencoders self-attention cross-attention masked tokens visible tokens decoder encoder representation learning ImageNet classification COCO instance segmentation